Leveraging Collection-Wide Similarities for Unsupervised Document Structure Extraction (2024)

Abstract: Document collections of various domains, e.g., legal, medical, or financial, often share some underlying collection-wide structure, which captures information that can aid both human users and structure-aware models. We propose to identify the typical structure of document within a collection, which requires to capture recurring topics across the collection, while abstracting over arbitrary header paraphrases, and ground each topic to respective document locations. These requirements pose several challenges: headers that mark recurring topics frequently differ in phrasing, certain section headers are unique to individual documents and do not reflect the typical structure, and the order of topics can vary between documents. Subsequently, we develop an unsupervised graphbased method which leverages both interand intra-document similarities, to extract the underlying collection-wide structure. Our evaluations on three diverse domains in both English and Hebrew indicate that our method extracts meaningful collection-wide structure, and we hope that future work will leverage our method for multi-document applications and structureaware models.1

heading: 1 Introduction
text: Knowing the structure of a typical document within a collection can be useful in various use cases across different domains. For example, consider the legal domain, where lawyers seek to analyze corpora of legal proceedings, looking for trends and patterns over time. In a retrieval scenario, they may look for punishment trends over many verdict decisions (Wenger et al., 2021). While each verdict decision normally includes a dedicated punishment section, it is often hard to locate it, because it is not consistently marked – different verdicts may
1Our code and data are available at https://github. com/SLAB-NLP/Doc-Structure-Parser
use different headers (“Punishment”, “Sentencing Decision”, or “Incurred Penalty”), and position the section in different document locations, requiring scholars to scan vast amounts of text. In an exploratory use case, lawyers may not have a welldefined apriori question, but instead interested in emerging collection-wide trends, for example, analyzing what judges take into account in verdict.
Furthermore, a collection-wide signal about document structure can also be leveraged by structureaware models for multi-document downstream applications, e.g., infusing the document structure as part of the Transformer architecture, to improve multi-document applications (Liu and Lapata, 2019; Zhang et al., 2023).
To support both human users and structure-aware models, we propose to identify the typical document structure within the collection (§2). This requires to capture recurring topics across the collection, while abstracting over arbitrary header paraphrases, and ground each topic to respective document locations. For example, in Figure 1 we want to identify “Case Summary”, “Evidence Presented” and “Verdict” as the recurring and prominent topics of a typical document within the collection of legal verdict decisions, as opposed to “Software Development Standards Implications”, which is document specific. This also requires to identify that “Verdict” and “Judgement Decision” represent the same topic, i.e., abstract over header paraphrases. The colored bounding boxes in Figure 1 represent the grounding unto each document.
Automatically extracting the typical document structure is challenging. While topic boundaries are often loosely defined by explicit headers, it is hard to use them directly to understand collection-level properties, as headers indicating the same information often vary in phrasing, for example “Verdict”, “Judgement Outcome” and “Decision”. Moreover, some section headers are local to individual document, and do not participate in the global structure.
9538
For example, “Software Development Standards Implication” in Figure 1. Finally, while section order provides some signal, it is often inconsistent, and may vary across different documents. The challenge is then align section headers across the collection, while being flexible enough to discard sections that do not reflect a global shared structure.
Following, we devise an unsupervised method using a collection-wide signal to perform the structure extraction (§3). To achieve this, we represent the document collection using a complete undirected weighted graph, whose nodes represent topic boundary candidates, and the weight of edges between each pair of nodes represents their semantic similarity. Such formulation allows us to model relations both within document, as well as across documents. For example, set large edge weight between “Case Overview” and “Introduction to the Case”, as they convey semantically similar topics. Finally, we find communities within the graph, where each community is a group of similar nodes that form a single component of the collection-wide structure, and filter the communities that best outline the typical document structure, which we call the collection-wide table of contents (ToC).
To illustrate the robustness of our method across various domains and languages, we curate three distinct datasets (§4). These include two English datasets from different domains (financial and legal), and a Hebrew dataset, composed of legal documents. This variety shows that our method is adaptive and effective under different linguistic and domain-specific contexts.
To evaluate our model, we propose three distinct metrics (§5). First is the “header intrusion”
human evaluation, which is adapted from the popular “word intrusion” metric for clustering evaluation (Chang et al., 2009), to evaluate the quality of the representation of the collection. Second, an automatic evaluation for the document-level grounding, to evaluate the predicted grounding coverage. Last, a qualitative analysis over the predicted ToC entries, manually exploring how meaningful are the produced entries, comparing them to an existing ToC of the financial documents collection.
We find that our method extracts meaningful typical document structure, capable of retrieving both the overall collection structure, while also mapping it unto individual documents, and that it is robust to varying domains and languages, with no supervision and little domain-specific adaptation. Detailed error analysis shows that while it is capable of mapping most topics unto documents, it struggles with identifying the exact topic boundaries.
Our released code allows future work to leverage collection-wide structure to enrich both user-facing applications, and to integrate document structure within LLMs to cope with multiple documents.
Our main contributions are: (1) we formally define a novel task of identifying the typical document structure from a document collection, (2) we curate three datasets for the task from diverse domains and languages, and (3) we develop an unsupervised approach for the task, by leveraging collection-wide signal in a community detection algorithm.

heading: 2 Task Definition
text: Given a collection of documents which share an underlying latent structure, we aim to recover a
structure which satisfies the three following requirements. First, we would like to list the most dominant topics which appear throughout the collection, and omit topics which appear only in few documents. This allows users to readily observe overall trends within the collection, without reading it in its entirety. Second, the representation should abstract over header paraphrases, allowing users to identify that certain sections discuss similar semantic topics, despite being phrased differently. Third, the structure should ground topics unto the documents in the collection, thus providing users with document-level structure, amenable for manual as well as automatic manipulation.
Formally, we denote a collection of documents D = {d1, . . . , dn}, where a model is required to output a tuple (T,M). T = {t1, . . . , tk} is a set of k most prominent topics across D. In the scope of this work, each topic ti should appear as a continuous text span in some document dj . For example, in Figure 1, T = {“Case Summary”, “Evidence Presented”, “Verdict”}. M : (T ×D) 7→ (N+ × N+) grounds topics to individual documents, mapping each topic to start and end indices. M is exemplified with colored bounding boxes in Figure 1.

heading: 3 Method
text: We describe our method for generating the ToC and grounding, which captures the typical document structure within a given document collection, and composed of three steps as outlined in Figure 2.

heading: 3.1 Identifying Document Headers
text: Following Erbs et al. (2013), in the scope of this work we assume document-level topics appear as explicit headers within documents. Later we show that this assumption holds in real-world domains.
Since header formatting may vary across domains, we consider their detection a corpus-specific task. We identify header candidates via rule-based heuristics, by leveraging a collection-wide signal that allows us to differentiate between certain patterns which are prominent throughout the collection (e.g., sentence length and capitalization patterns (Gutehrlé and Atanassova, 2022)), while filtering out other document elements which are sometimes styled as headers (e.g., page numbers, recurring signatures, copyright licensing, etc.).

heading: 3.2 Building a Collection Representation
text: Next, we face a collection-level challenge. After segmenting each document to topics, it is not clear how to represent both intra- and inter-document similarities across the entire collection. Furthermore, the similarity between topics hinges on a myriad of factors, including semantic similarities between the topic headers, their content, and their location within the document. For example, in Figure 2 we want to identify that “Verdict” in one document and “Judgement Desicion” in another are both related to a shared ‘Verdict’ topic, which is dominant in legal documents, and usually located towards the end of the verdict decision.
To model the different intra- and inter-document
similarities, we represent a document collection using a complete undirected weighted graph G = (V,E) as elaborated below.
Graph nodes. The nodes in the graph include all headers identified previously. Each node v ∈ V (G) is associated with a 4-tuple: {doc_ind, seg_ind, head_text, body_text}. doc_ind ∈ N denotes the index of the document within the collection, seg_ind ∈ [0, 1] denotes the normalized sequential position within the document, while head_text and body_text are the two continuous text spans of the header and following body text. For example, one of the nodes in Figure 2 represents the segment at the 0.5 normalized position from the 3rd document, its header is “Judgement Outcome” and its body text starts with “The court..”.
Edge weights. As described above, G is a complete graph over all identified topic boundary candidates from documents within the collection. For each edge (v1, v2), we define weight(v1, v2) ∈ R as a weighted sum of three similarity metrics:
weight(v1, v2) = λhead · head_sim(v1, v2) + λbody · body_sim(v1, v2) + λpos · pos_sim(v1, v2)
(1)
Where head_sim(v1, v2) ∈ R represents the similarity between the headers corresponding to the two nodes, and body_sim(v1, v2) ∈ R represents the similarity between their bodies. We compute both using cosine similarities over a language model embedding. The pos_sim(v1, v2) ∈ R similarity metric computes the similarity of ordering within the document, which encodes the assumption that documents sometimes follow similar topic ordering. In particular, we define pos_sim as follows:
pos_sim(v1, v2) =
(|seg_ind(v1)− seg_ind(v2)|)−1 (2)
Hence, pos_sim is larger the more the respective segments are in similar positions within their respective documents.
Finally, λhead, λbody, λpos are hyperparameters, non-negative and sum up to 1, weighting the three metrics according to different corpus characteristics. For example, if the document collection follows a strict order across all documents, λpos could be large, as opposed to the case that the order is not strict but the headers themselves are similar, and then λhead would be larger.

heading: 3.3 Extracting the Typical Document Structure
text: Finally, we want to find the the most representative collection-wide ToC and its corresponding grounding unto each document. At this step we want to find list of important topics which appear in many documents throughout the collection. In addition, we are interested in mapping the list of important topics unto each document in the collection.
As we outlined in Section 2, a good output would consist of coherent topics (e.g., all topics mapped to "Case Summary" would discuss the facts of the case), covering as many documents in the collections, and much of each individual document.
We achieve this by finding communities in the graph. Each community is a group of nodes heavily connected to each other by high-weight edges and lightly connected to nodes outside the community by low-weight edges. Ideally, the inter-community edges represent high similarity between nodes, and hence the community represents a coherent topic across different documents within the collection.
For community detection, we apply the Louvain algorithm (Blondel et al., 2008). We use this algorithm as it has a small number of hyper-parameters compared to other community detection algorithms, and it has shown reliable results over various NLP tasks (Lucy et al., 2023). The objective of the Louvain algorithm is to maximize modularity, which correlates with the ratio between edges inside the community to edges outside the community.
Finally, we generate the collection-wide ToC out of the best communities. We find a subset of k communities that maximize the coverage of the entire collection, which indicates that the chosen communities represent the topics that are relevant to many of the documents and covers the most of each document. For example, in Figure 1 we omit “Software Development Standards Implications” as it appears only in a single document.
Formally, let C represent the set of all communities output by Louvain, where each community c ∈ C is a list of nodes. We want to find the subset S ⊂ C, containing k communities, that maximizes the coverage of each individual document d across the entire collection D:
argmax S⊂C s.t. |S|=k
∑
d∈D
[ 1
|d| ∑
v∈d 1[∃c∈S s.t. v∈c]
] (3)
To form the k ToC entries, we then find the centroid of each of the k best communities, and ground
Dataset Name # docs # words Domain Language Strict structure Preserve sections order Headers similarity
each representative centroid to the text spans of its corresponding community.

heading: 4 Data Curation
text: To evaluate our method, we curate three document collections. As presented in Table 1, these collections cover different domains and languages, with varying structure properties, showing the robustness of our method.

heading: 4.1 Form-10K
text: Form-10K is an annual financial report required by U.S. Securities and Exchange Commission, summarizing a company’s annual financial performance. It includes information such as company history, organizational structure, executive compensation, equity, and audited financial statements.
The Form-10K files were collected from the SEC EDGAR platform.2 We sampled 500 company CIK tickers from a list of public companies, and used these as the basis for EDGAR form retrieval. We then extracted the document’s texts by cleaning the markup and filtering non-textual lines (e.g. tables) using regular expressions.

heading: 4.2 CUAD
text: Hendrycks et al. (2021) presented Contract Understanding Atticus Dataset (CUAD), consisting of 510 commercial legal contracts. This dataset was curated to evaluate the extraction of key clauses in contracts, providing labeled clauses that were manually extracted from the documents. Since we are only interested in predicting the structure of the documents, we ignore the provided labels.
To run our method over this dataset, we converted the provided pdf files to raw texts, resulting in raw texts of 389 legal contracts.3
In contrast with Form-10K, CUAD consists of different types of contracts, such as affiliate agree-
2Using the secedgar python package https://github. com/sec-edgar/sec-edgar
3We used Adobe API package to convert pdf to text https://developer.adobe.com/document-services/ docs/overview/pdf-extract-api/
ments, license agreements, marketing, or manufacturing. Hence, the contracts may diverge in their structure, content and order.

heading: 4.3 Hebrew Verdict Decisions
text: We use Habba et al. (2023)’s curated dataset of verdict decisions from the Israeli court, in Hebrew. The original dataset consists of 855 verdict decisions in sexual offense cases, which we then filtered manually, keeping only the documents with explicit headers, resulting in a collection of 277 Hebrew verdict decisions.
Since all verdict decisions in this dataset are of the same domain (sexual assault cases), we expect them to contain some overlapping topics and highlevel structure similarities. But, since there is no pre-defined structure and each judge has their personal style, there are also expected differences and some parts may be added or omitted.

heading: 5 Results
text: We evaluate three different aspects of our method, including human evaluation of the communities we produce (§5.2), document-level evaluation of the ToC grounding (§5.3), and a qualitative analysis of the predicted ToC (§5.4).

heading: 5.1 Experimental Setup
text: Our architecture introduces several hyperparmeters designed to capture different datasets properties. Below we outline their configuration for each of our three collections. These are summarized in Table 1, and presented in detail in Table 4 in the Appendix.
First, our method embeds the document texts using a pre-trained language model. For the two English datasets we use a version of the MPNet language model (Song et al., 2020), as it was reported with the highest average performance on sentence embedding and semantic search tasks.4,5 For our
4The MPNet version we use https://huggingface.co/ sentence-transformers/all-mpnet-base-v2
5Performance of different pre-trained English LMs https: //www.sbert.net/docs/pretrained_models.html
Hebrew Verdict Decisions dataset we use a version of AlephBERT (Seker et al., 2021).6
Second, we configure the weighting of the different similarity metrics in Equation 1, according to apriori knowledge of the domains. For example, we set a high header similarity weight (λhead) for the Form-10K corpus, to reflect the expectation for very similar headers across documents. In contrast, the structure of the documents in CUAD is more flexible, so we set a more uniform weighting.

heading: 5.2 Graph Representation Evaluation
text: In this evaluation, participants are shown 10 headers, 9 of which are chosen headers from the same community at random, and the last header is chosen at random from outside the community. Participants are then tasked with identifying the intruder.
This task assesses if our predicted communities are well separated. We expect meaningful communities to demonstrate high internal similarity and low similarity with other communities. Therefore, if our communities are well-defined and meaningful, identifying intrusions should be straightforward, as they will noticeably differ from the rest of the community.
Our evaluation is an adapted version of the “word intrusion” task (Chang et al., 2009), widely used for human evaluation of topic clustering (Bhatia et al., 2017, 2018; Prouteau et al., 2022). We adapt this metric to account for the higher variability presents in headers as opposed to single-word topics by allowing annotators to choose an arbitrary number of intruders. The number of options they mark is reflected in a confidence metric:
conf = 1− num_marked− 1 num_options
(4)
Intuitively, the more intruders that the annotator marks, the less they are certain of their annotation. At the extremes, conf = 1 when an annotator marks a single intruder, and conf = 110 if they mark all options as intruders. This metric does not account for the accuracy of the annotation, which is calculated separately, if either any of the marked options is the actual intruder.
Crowdsourcing configuration. We run this evaluation through Amazon Mechanical Turk, with a pool of 12 participants, where each unique sample was annotated by exactly one participant, for a total
6The AlephBERT version we use https://huggingface. co/imvladikon/sentence-transformers-alephbert
of 900 collected annotations (around 450 samples for each English dataset). We pay 0.1-0.2 USD for a single annotation, aiming for an hourly pay of 12 USD. To ensure quality annotations, we start with a qualification test where participants need to reach accuracy above 0.4 with confidence above 0.25 over a few examples of our method’s output. In Figure 5 in the Appendix we provide an example of the annotation interface. As for the Hebrew Verdict Decisions, we collect 132 annotations via a group of 8 in-house Hebrew-speaking graduate students, who show similar performance to MTurk qualified workers.

heading: 5.2.1 Key Findings
text: In Table 2 we present the results for the header intrusion human evaluation, and highlight key findings below.
Our method predicts a meaningful collection representation on par with traditional topic modelling. In Table 2 we see that the accuracy for intrusion detection is more than double the chance accuracy for the observed ∼80% confidence. Moreover, we find that our reported accuracy falls inside Chang et al. (2009)’s interquartile range (IQR) of word-intrusion results.
Our method performs better on document collections with a more strict nature of structure. As seen in Table 2, accuracy is highest for the Form10K collection, which may attributed to its stricter structure, compared to our other two collections.
Our method sometimes conflates opposite headers from opposite topics. We manually analyze
the Hebrew Verdict Decisions annotations, finding that some of the low confidence annotations (i.e., when annotators mark many options for the intruder) happen when our method conflates opposite topics in the same community. For example, “defense arguments” and “prosecution arguments” are wrongly clustered in the same community.7 Considering opposite topics as close in the embedding space is a known property of language models (Vahtola et al., 2022).

heading: 5.3 Document Grounding Evaluation
text: In a second evaluation effort, we aim to quantify the quality of our method’s automatic grounding from collection-wide ToC topics unto text spans in individual documents.
Annotating gold labels. To evaluate our method’s grounding performance, we annotate some of our unsupervised data in two of our domains. The Form-10K dataset follows a strict structure, so we apply simple heuristics to automatically generate gold label grounding, resulting in a collection of 266 documents with grounding gold labels.8 In contrast, for the Hebrew Verdict Decisions we use human annotations to generate its grounding gold labels, done by a domain expert (Hebrew speaking law student). Each manual annotation took approximately 45 minutes, with a total of 15 labeled documents. A screenshot of the annotation interface is provided in Figure 4 in the Appendix. We omitted CUAD from this particular evaluation due to budget constraints.
Metrics. We formulate exact and partial match scores, evaluating the overlap between the gold and the predicted grounding. Partial match credits any overlap between the text spans in predicted
7All examples from Hebrew Verdict Decisions are translated from Hebrew.
8Gold ToC for Form-10K is based on https://en. wikipedia.org/wiki/Form_10-K
and gold, while exact match counts only perfectly aligned predictions. The formal mathematical definitions are presented in Appendix B.1.
Baselines. To the best of our knowledge, we present the first method which aims to identify ToC at the document-collection level, and hence we are not aware of applicable baselines. Instead, to gauge the difficulty of this task we include two naive approaches. First, a baseline predicting most frequent class in the gold annotations. For the Form-10K corpus this is “Exhibits, Financial Statement Schedules” (22% of headers) and for Hebrew Verdict Decisions this is “prosecution evidence” (20% of headers). The second baseline predicts each gold ToC header uniformly at random, averaged over 100 different random seeds.

heading: 5.3.1 Key Findings
text: The results for our two metrics and two document collections are presented in Table 3. Several observations can be drawn based on these results.
Our method captures the correct structure of a document. The high partial match scores indicate that our method is able to predict the correct document structure, as almost every gold topic grounding has some overlap with the predicted topic grounding (Micro F1 score of 92.6). As shown in Table 3, our method outperforms the two naive baselines in all F1 scores.
Exact match score is noisy. Even if only a single sentence is separating between a gold and a predicted topic grounding boundaries, the exact match score for the topic will be 0. We find this score very strict, and somewhat arbitrary. In addition, topics related to longer text spans, are more probable to reach a low exact match score. This is indicated by the first three rows in Table 5 in the Appendix, representing the three largest classes, showing lower exact match scores than all other classes.

heading: 5.4 Qualitative ToC Analysis
text: Lastly, we suggest a collection-level qualitative evaluation, to explore the quality of the output ToC. Below we provide several key observations derived from our manual mapping between the predicted and gold Form-10K ToC entries, visualized in Figure 3.
We were able to map 14 out of the 15 gold ToC entries to the predicted ToC entries. The only gold entry that was not mapped is “8. financial statements”. After manually exploring the ToC grounding, we find that our method sometimes confuses this topic with “6. Consolidated Financial Data”.
The 14 gold ToC entries align with 19 predicted entries, in the following manner:
• 13 predicted ToC entries that were textually almost identical to the gold entries, with a straight-forward mapping.
• 1 mapping which followed a manual exploration of the dataset – we mapped “4. Mine Safety Disclouser” to “Submission Of Matters To A Vote Of Security Holders”, as we find
that in many of the documents the latter is the actual header of the 4th topic.
• 5 subsection headers (‘A Risk Factors’, ‘B Unresolved Staff Comments’, etc.), that we map to their gold topic headers, as our method predicts a flat structure while the Form-10K consists of an hierarchical structure.
Beyond the 19 predicted entries that align with gold entries, our method predicted additional 12 entries. Some topics indeed recur throughout the collection and accordingly receive high coverage scores, e.g., “Service Providers” or “Interest Income”, and may be considered too granular. Others, attributed to noise, e.g., “Total”, “Table” or “Yes X”, receive low coverage scores, and their emergence as topic candidates could stem from degraded performance at the header detection step.

heading: 6 Related Work
text: A seminal theory of a pragmatic meaning is rhetorical structure theory (RST Mann and Thompson, 1988), that inspired much follow-up work, mostly approaching document structure extraction as a
supervised task (Arnold et al., 2019; Kang et al., 2022; Xia et al., 2022; Hua and Wang, 2022). The need for supervision narrows the application of such tools only to high-resource languages and small number of domains. Our method is unsupervised, does not require labeling, and is applicable to many languages and domains.
Another line of work suggest to approach structure extraction as an unsupervised task, to extend its applicability (Xing and Carenini, 2021; Born et al., 2022; Li et al., 2023), but they still approach this as a single-document task. In our work we take the entire collection as our input, leveraging signal regarding the underlying collection-wide structure.
Xing et al. (2022) discovered that injecting external (above-sentence) information can help with identifying discourse dependency structure, but they only look at individual documents. We extend this observation into modeling relations across an entire collection, to inject external information about the structure components of the collection.
Finally, theories like CST (Radev, 2000) and CAST (Altmami and Menai, 2020), or (Lu et al., 2018), suggest to look at the entire data collection to extract the main components that capture a typical structure layout of a document within the collection, but, their approach too relies on supervision and labeled data.

heading: 7 Conclusion
text: In this paper we suggested an unsupervised method to extract the typical document structure within a collection, leveraging similarities across documents that come from the same collection with some shared underlying latent structure.
We evaluated the extracted typical document structure through three different evaluation metrics, over three different datasets, showing our method’s robustness to different domains and languages. The evaluations indicated that our method succeeds in identifying the different topics consisting the typical document structure, while showing room for improvement in its accuracy to detect the exact location of where a topic starts and where it ends.
Our method can be leveraged by users of multidocument applications, allowing them more focused browsing over a collection of documents, which can be useful for downstream tasks like retrieval and summarization. As for models, future work can utilize our method and inject the collection-wide structure to structure-aware mod-
els, to improve downstream applications like multidocument summarization or cross document information extraction.

heading: Acknowledgements
text: We would like to thank the anonymous reviewers for their helpful comments. Members of the SLAB group at the Hebrew university for their productive discussion. This research was supported by grants from Allen Institute for AI, the Israeli Ministry of Science and Technology (grant no. 2088), and the Council for Higher Education.
Limitations
Currently, our method relies on segmentation induced by the headers in the document, which limits our work only applicable on documents with explicit headers. Future work can extend our collection-wide approach to develop a multidocument unsupervised method to induce segmentation for document collections without explicit headers.
Furthermore, our method only predicts a flat list of the prominent topics, and does not express the hierarchical nature of structure that might be present. Future work can extend our method to predict also hierarchical structure.
Ethics Statement
All annotators were informed ahead that their annotations are curated for research purposes, and how the data is going to be used.
The Hebrew Verdict Decisions dataset contains sensitive information pertaining to sexual harassment offenses. The original dataset we utilized underwent a rigorous anonymization process to protect individual privacy. We do not publicly publish this dataset, and accessing it requires authorization from the dataset’s original proprietors (Habba et al., 2023).
Recognizing the potential for distress by annotating the Hebrew Verdict Decisions, we issued a trigger warning for all annotation tasks and obtained explicit consent from annotators before their participation. This ensured they were fully informed of the risks associated with exposure to sensitive and potentially triggering content.

heading: A Parameters For Evaluation
text: As mentioned in Section 5.1, below we describe the different parameters used for running the evalution.
Headers detection. In our implementation we detect headers for each dataset with a different set of heuristics. For example, for the Form-10k dataset, we search for sentences shorter than 10 tokens, with capitalization over > 50% of the words in the sentence. This produces high recall (most headers are caught through this heuristic), but low precision (many of the caught header candidates are not really headers). An example for a scenario where a sentence is wrongly marked as a header candidate is for missing sections, in which the only section content is ‘None’ or ‘Omitted’. Furthermore, some of the header candidates are real headers, but not informative ones, for example ‘Section A’. In such cases, we also drop these sentences from the set of header candidates using regex.

heading: B Additional Evaluation Analysis
text: B.1 Exact and Partial Match Scores
Topic boundaries are a set of consecutive text spans that are labeled with the same ToC header. Following, we define an exact match of a topic grounding if the text spans forming the gold label topic grounding, match perfectly with the text spans that are predicted with that header, while for partial match it is enough if there is one text span within that topic boundaries that was predicted correctly.
These two matching scores are formalized as follows: let s(w, d) be the range of text spans in document d, that are labeled as class w. So, we evaluate the intersection between sgold(w, d) and spred(w, d). Full match precision is if spred(w, d)\ sgold(w, d) = ∅, i.e., all text spans in d that were predicted w are labeled correctly, and full match recall is if sgold(w, d)\spred = ∅, i.e., all text spans in d that are labeled w are covered by the correct predictions. For partial match it is enough to have a non-empty intersection between sgold(w, d) and spred(w, d).
B.2 Document-Level Grounding For Form-10K
In Table 5 we provide our method’s detailed results for each ToC entry’s grounding, i.e., precision, recall and F1 exact and partial scores, over the Form-10K dataset.
B.3 Mapping Between Predicted and Gold Form-10K Conceptual ToC
In Figure 3 we provide the manually mapping between the predicted Form-10K ToC entries, to its gold label ToC.

heading: C Human Annotations
text: C.1 Annotating Gold Labels For Grounding
Figure 4 is a screenshot of the annotation interface for grounding gold labels. We use this interface to annotate the Hebrew Verdict Decisions dataset, but for readability we provide here the interface over a document from CUAD.
C.2 Header Intrusion Task
Figure 5 is a screenshot of the annotation interface for the header intrusion evaluation, As described in Section 5.2. Annotators were chosen from Englishspeaking countries: USA, Canada, and UK. Mturk
does not provide further demographic information about annotators.

heading: Future Work
text: Future work can extend our collection-wide approach to develop a multidocument unsupervised method to induce segmentation for document collections without explicit headers. Furthermore, our method only predicts a flat list of the prominent topics, and does not express the hierarchical nature of structure that might be present. Future work can extend our method to predict also hierarchical structure