A Scalable Framework for Table of Contents Extraction from Complex ESG Annual Reports
Authors: Xinyu Wang
Lin Gui
Yulan He
Venue: EMNLP 2023 Main (2023)
Keywords: Table of Contents Extraction
Tree
TL;DR: Scalable Framework for Table of Contents Extraction from ESG Reports

Abstract: Table of contents (ToC) extraction centres on structuring documents in a hierarchical manner. In this paper, we propose a new dataset, ESGDoc, comprising 1,093 ESG annual reports from 563 companies spanning from 2001 to 2022. These reports pose significant challenges due to their diverse structures and extensive length. To address these challenges, we propose a new framework for Toc extraction, consisting of three steps: (1) Constructing an initial tree of text blocks based on reading order and font sizes; (2) Modelling each tree node (or text block) independently by considering its contextual information captured in node-centric subtree; (3) Modifying the original tree by taking appropriate action on each tree node (Keep, Delete, or Move). This construction-modelling-modification (CMM) process offers several benefits. It eliminates the need for pairwise modelling of section headings as in previous approaches, making document segmentation practically feasible. By incorporating structured information, each section heading can leverage both local and long-distance context relevant to itself. Experimental results show that our approach outperforms the previous state-of-the-art baseline with a fraction of running time. Our framework proves its scalability by effectively handling documents of any length.

A Scalable Framework for Table of Contents Extraction from Complex ESG Annual Reports
Xinyu Wang Lin Gui', Yulan Hel
Department of Computer Science, University of Warwick Department of Informatics, King's College London
'The Alan Turing Institute
Xinyu. Wang. 11 11@warwicck ac. uk 1lin.1.gum yulan. hew.hheekkkce. ac. uk
Abstract
and organisation. While traditional summarisation offers concise representation of document's content, a Table of Contents (ToC) presents a struc- tured and hierarchical summary. This structural organisation in a ToC provides a comprehensive pathway for pinpointing specific information. For example, when seeking information about a com- pany's ' carbon dioxide emissions, ToC enables systematic navigation through the information hierarchy. In contrast, conventional summarisa- tion might only provide a vague indication of such information, requiring sifting through the entire document for precise detail.
Table of contents (ToC) extraction centres on structuring documents in a hierarchical man- ner. In this paper, we propose a new dataset, ESGDoc, comprising 1,093 ESG annual reports from 563 companies spanning from 2001 to 2022. These reports pose significant challenges due to their diverse structures and extensive length. To address these challenges, we pro- pose a new framework for Toc extraction, con- sisting of three steps: (1) Constructing initial tree of text blocks based on reading order and font sizes; (2) Modelling each tree node (or text block) independently by considering its con- textual information captured in node-centric subtree; (3) Modifying the original tree by tak- ing appropriate action on each tree node (Keep, Delete, or Move). This construction coustruciionmoddellngg modification (CMM) process offers several ben- efits. It eliminates the need for pairwise mod- elling of section headings as in previous ap- proaches, making document segmentation prac- tically feasible. By incorporating structured information, each section heading can leverage both local and long-distance context relevant to itself. Experimental results show that our approach outperforms the previous state-of-the- art baseline with a fraction of running time. Our framework proves its scalability by effectively handling documents of any length.
Several datasets have been proposed to facilitate the research in document understanding (Zhong et al., 2019b; Li et al., 2020; Pfitzmann et al., 2022). Most of these studies lack structured construc- tion of documents and primarily focus on well- structured scientific papers. dataset called Hier- Doc (Hierarchical academic Document) (Hu et al., 2022) was introduced to facilitate the development of methods for extracting the table of contents (ToC) from documents. This dataset was compiled from scientific papers downloaded from 2 which typically short and well-structured. The hierarchical structure often be inferred directly from the headings themselves. For example, the heading "1. Introduction" can be easily identified as a first-level heading based on the section number- ing. Moreover, due to the relatively short length of scientific papers, it is feasible to process the entire document as whole. Hu et al. (2022) proposed the multimodal tree decoder (MTD) for ToC extrac- tion from HierDoc. MTD first utilises text, visual, and layout information encode text blocks identi- fied by a PDF parser; then classifies all text blocks into two categories, headings and non-headings; and finally predicts the relationship of each pair of headings, facilitating the parsing of these headings into tree structure representing ToC.
1 Introduction
A considerable amount of research has been pro- posed to comprehend documents (Xu et al., 2019; Zhang et al., 2021; Xu et al., 2021a,b; Peng et al., 2022; Li et al., 2022; Gu et al., 2022; Shen et al., 2022; Lee et al., 2022, 2023) which typically involves the classification of different parts of document such title, caption, table, footer, and on. However, such prevailing classification of- ten centres on a document's 's local layout structure, sidelining a holistic comprehension of its content
'vaillale at http://gitub..om/knnnwwgggg
2nttps://arxiv.orr

---

Diverse, Engaged and Capable Workforce
OCCIPITINAL
$5,300m
$2 4.4mm
$1.812m
H+T 210000 EMPLOYEES
Figure 1: Five examples of ESG reports, with the left three presented in portrait orientation and the right-most two in landscape orientation. They show wide range of diverse structures. It is common to observe the absence of section numbering.
However, understanding long documents such ESG (Environmental, Social, and Governance) annual reports poses significant challenges com- pared to commonly used scientific papers. First, ESG reports tend to be extensive, often exceed- ing 100 pages, which is uncommon for scientific papers. Second, while scientific papers generally adhere to standard structure that includes abstract, introduction, methods, results, discussion, and con- clusion sections, ESG reports exhibit more diverse structures with a wide range of font types and sizes, Third, ESG reports often include visual elements such as charts, graphs, tables, and infographies to present data and key findings in a visually appeal- ing manner, which adds complexity to the docu- ment parsing process. Some example ESG reports are illustrated in Figure 1.
In order to overcome the challenges presented in ESGDoc, propose new scalable framework, consisting of three main steps: (1) Constructing an initial tree of text blocks based on reading order and font sizes; (2) Modelling each tree node (or text block) independently by considering its contex- tual information captured in node-centric subtree; (3) Modifying the original tree by taking appro- priate action on each tree node (Keep, Delete, or Move). Our method is named CMM (Construction- Modelling- Modification).
This approach allows higher-level headings to focus on capturing high-level and long-distance in- formation, while lower-level headings focus more on local information. Additionally, CMM also mod- els each heading independently, removing the need for modelling pairwise relationships among head- ings and enabling more effective document seg- mentation. Here, we can divide documents based on the tree structure instead of relying on page di- visions. This ensures that each segment maintains both local and long-distance relationships, preserv- ing the long-distance connections that would be lost if division were based page boundaries. As CMM does not require the processing of document as whole, it can be easily scaled deal with lengthy documents. Experimental results show that our approach outperforms the previous state-of-the- art baseline with only a fraction of running time, verifying the scalability of our model as it is ap- plicable to documents of any length. Our main contributions are summarised as follows:
In this paper, we develop new dataset, ESGDoc, collected from public ESG annual reports from 563 companies spanning from 2001 to 2022 for the task of ToC extraction. The existing approach, MTD (Hu et al., 2022), faces difficulties when deal- ing with challenges presented in ESGDoc. MTD models relationships of every possible heading pairs and thus requires the processing of the entire document simultaneously, making it impractical for lengthy documents. As will be discussed in our experiments section, MTD run into out-of-memon issue when processing some lengthy documents in ESGDoc. Moreover, MTD only uses Gated Recur- rent Unit (GRU) (Cho et al., 2014) to capture the context of a section heading, lacking din-tdit interaction, particularly for high-level headings that may be tens of pages apart.
We introduce new dataset, ESGDoc, com- prising 1,093 ESG annual reports specifically designed for table of contents extraction.
/ http://re.pposbiiiiiyrrrrr.....

---

We propose a novel framework that pro- cesses documents in a ceasstructio-modellinn modification manner, allowing for the decou- pling of each heading, preserving both local and long-distance relationships, and incorpo- rating structured information.
iri and Jain, 2007: Doucet et al., 2011). Tuarob et al. (2015) designed some features and use Ran- dom Forest (Breiman, 2001) and Support Vector Machine (Bishop and Nasrabadi, 2006) to predict section headings. Mysore Gopinath et al. (2018) propose system for section titles separation. MTD (Hu et al., 2022) represents a more recent approach, fusing text, visual, and layout information to detect section headings from scientific papers in the Hi- erDoc (Hu et al., 2022) dataset. It also uses GRU (Cho et al., 2014) and attention mechanism to clas- sify the relationships between headings, generating the tree of ToC. While MTD performs well on Hi- erDoc, it requires modelling all headings in the entire document simultaneously, which is impracti- cal for long documents. To address this limitation, propose a framework that decouples the relationships of headings for ToC extraction and introduces more structural information by utilis- ing font size and reading order, offering a more practical solution for long documents.
We present novel graph-based method for document segmentation and modelling, en- abling the retention of both local and long- distance information within each segment.
2 Related Work
Datasets Many datasets have been proposed for document understanding. PubLayNet dataset (Zhong et al., 2019b) is a large-scale dataset col- lected from PubMed Central Open Access, which uses scientific papers in PDF and XML versions for automatic document layout identification and anno- tation. Aaticos-regions dataset (Soto and Yoo, 2019) offers more consistent and tight-fitting annotations. DocBank dataset (Li et al., 2020) leverages the la- tex source files and font colour to automatically an- notate vast number of scientific papers from arviv. DocLayNet dataset (Pfitzmann et al., 2022) extends the scope from scientific papers other types of documents. However, these datasets primarily con- tain annotations of the type and bounding box of each text, such as title, caption, table, and figure, but lack structured information of documents.
3 Dataset Construction
To tackle the more challenging task of ToC extrac- tion from complex ESG reports, construct new dataset, ESGDoc, from Responsibility Reports. com".. Initially, we have downloaded 10,639 reports in the PDF format. However, only less than 2,000 reports have ToC in their original reports. To facilitate the development of an automated method for ToC ex- traction from ESG reports, we selectively retrain reports that already possess a ToC. The existing ToC serves the reference label for each ESG re- port, while the report with the ToC removed is used for training framework specifically designed for ToC extraction.
Approaches for Document Understanding In terms of methods for document understanding, a common approach is the fusion of text, visual, and layout features (Xu et al., 2019; Zhang et al., 2021; Xu et al., 2021a,b; Peng et al., 2022; Li et al., 2022), where visual features represent images of texts and the document, and layout features comprise bound- ing box positions of texts. Some methods also introduced additional features. For instance, XY- LayoutLM (Gu et al., 2022) incorporates the read- ing order, VILA (Shen et al., 2022) utilises visual layout group. FormNet (Lee et al., 2022, 2023) employs graph learning and contrastive learning. The aforementioned methods focus classifying individual parts of the document rather than under- standing the structure of the entire document.
Our final dataset comprises 1,093 publicly avail- able ESG annual reports, sourced from 563 distinct companies, and spans the period from 2001 to 2022. The reports vary in length, ranging from 4 pages to 521 pages, with an average of 72 pages. In contrast, HierDoc (Hu et al., 2022) has total of 650 scien- tific papers, which have average of 19 pages in length. We randomly partitioned the dataset into a training set with 765 reports, a development set with 1 110 reports, and a test set with 218 reports.
Text content from ESG reports was extracted us- ing PyMuPDF5 in format referred to as "block". A block, defined as text object in the PDF stan-
Table of Contents (ToC) Extraction In addition document understanding, some work has been conducted on the extraction of ToC. Early meth- ods primarily relied on manually designed rules to extract the structure of documents (Nambood-
httpss//ww..rsponiilyyyyeerrrr.. https://pppmmpf.feeeeeeeeeeeeeee

---

dard, which is usually paragraph, can encompass multiple lines of text. We assume that the text within a text object is coherent and should be inter- preted as a cohesive unit. Each block comprises the following elements: text content, font, size, colour, position, and id. The id is a unique identifier as- signed to each block to distinguish blocks that con- tain identical text content. The position refers to the position of the block within page in the ESG PDF report, represented by four coordinates that denote the top-left and bottom-right points of the block bounding box. Other clements, such as font, size, and colour, provide additional information about the text.
learn a function which determines the operation ('Keep', 'Delete', or 'Move') to be performed on the node. In order to enable document segmenta- tion and capture the contextual information relating to the node, our approach involves extracting a sub- tree encompassing its neighbourhood including the parent, children and siblings, within a range of nd hops. Subsequently, works (GATs) (Brody et al., 2021; Velickovic al., 2017) to update the node information within the subtree. An overview of our proposed framework is illustrated in Figure 2.
use Graph Attention Net-
Before delving into the detail of our proposed framework, we first define some notations relating to node operations. PA(x;) as the parent node of node 24, PR(x.) as the preceding sibling node of T4. SU;);) the subsequent sibling node of
4 Methodology
We propose a framework for ToC extraction based on the following assumptions:
3i. We also define (R)(x)) as all the preceding sibling nodes of Zip-
Assumption 1 Humans typically read documents a teft-to-right, top-to-bottom order, and a higher- level heading is read before its corresponding sub- heading and body text.
4.1 Tree Construction
We first construct a complete tree T,, consisting of all identified blocks using PyMuPDF, based on reading order and font sizes. For each node 3; we find a node in its previous nodes žj E { } that is closest to m2 and 8j > 8;. Then xe becomes a child of æz. A detailed algorithm is in Appendix A. Following the principles outlined in Assumptions 1, 2 and 3, this approach assumes that the ToC is contained within the tree structure, shown in the top-left portion of Figure 2. The subsequent steps of our model involve modifying this tree to generate the ToC.
Assumption 2 In table of contents, the font size of higher-level heading is no smaller than that of a lower-level heading or body text.
Assumption 3 In a table of contents, headings of the same hierarchical level share the same font size.
In our task, document is defined a set of blocks. To replicate the reading order of humans, we reorder the blocks from the top-left to the bottom-right of the document. We employ the XY- cut algorithm (Ha al., 1995) to sort the blocks. The sorted blocks are denoted as {z J= where (r<s) precedes x; and {a>+ follows xy Here, ns represents the total number of blocks. For each block 24, we define 8; as its size.
4.2 Tree Modelling
In this section, for a given tree node, we aim to learn a function which takes the node representa- tion as input and generates the appropriate opera- tion for the node. In what follows, we first describe how we encode the contextual information of a node, and then present how to learn node represen- tations.
Problem Setup Given list of blocks, ToC ex- traction aims to generate a tree structure represent- ing the table of contents, where each node corre- sponds to a specific block æ. We introduce a pseudo root node r as the root of the ToC tree.
We propose to initially construct a full tree con- taining all the blocks, where the hierarchical rela- tion between blocks is simply determined by their respective font sizes. Specifically, when two blocks, 24 and wy. are read in sequence, if they are close each other and their font sizes s; > sy, then 2) becomes a child of z;. We then modify the tree by removing or rearranging nodes as necessary. Es- sentially, for a node (i.e., a block) I;, we need to
Node-Centric Subtree Extraction To effec- tively encode the contextual information of a tree node and to avoid processing the whole document in one go, we propose to extract node-centric sub- tree, t,, a tree consisting of neighbourhood nodes, including PR(a;) and (Uxx), of node 2;, ex- tracted via Breadth First Search (BFS) on d; with a depth ny. Here, na is a ppperr-arameter.. The neighbourhood nodes consist of the parent node,

---

Document
Reorder (reading order)
4.
3.2.1
111
Subtree
Initial Full Tree
Sibtroe Segmen
Subtroe
Tree Modification
Tree of Table of
Tree Madelling
Heading
Figure 2: Overview of CMM. Initially, blocks across multiple pages in a document shown in top-left, are reordered into sequence based reading order (Top-right). A full consisting of all text blocks is constructed based reading order and font size (Center-left). Subsequently, for each node, node-centric subtree is extracted and modelled by graph neural network (GNN). the subtree shown in center-right, the first-level heading can access both long-diin distance relationships with other first-level headings '2'' '3. '4.', and local relationships with heading '.7', "1.2", and bodies. In contrast, in the subtree shown in bottom-right, the second-level heading '3.2.' concentrates more on local information but also has access to some global information. When modelling with GNN, each node is connected with its neighbourhood nodes, including parent, children and siblings. After the tree modelling phase, the node-level operations (Keep, Delete, and Move) are predicted. The original tree is then modified according the node-level operations, resulting in the final Table of Contents (Bottom-left), The nodes numbered and coloured in this figure are for illustrative purposes; some body nodes are omitted for brevity. During inference, the model is unaware whether a node is heading or non- non-heading.
children nodes. and the sibling nodes of node 24, shown in the right portion of Figure 2. Apart from the edges linking parent and child, have additionally added edges connecting neighbouring sibling nodes.
points. The representation of is derived from text encoder and fi with Multilayer Perceptron (MLP) follows:
b, MLLP(TexxEEccodd((()..
(1)
Node Encoding Before discussing how to update node representations in a subtree, we first encode each node (or block) x; into vector representation. We employ a text encoder, which is a pre-trained language model, to encode the text content of Tzj We also utilise additional features from x; defined ft. These features include: (1) pdf page number; (2) font and font size; (3) colour as RGB; (4) the number of lines and length; and (6) the position the bounding box of the block, represented by the coordinates of the lef- and bottom-right
where by denotes the hidden representation of block 2; and [..-] denotes concatenation.
To simulate the human reading order, we apply one-layer bi-directional Gated Recurrent Unit
(GRU) (Cho et al., 2014) on nodes of the subtree with in-order traversal as follows:
u [tal GRU(t))
(2)
where { {u:} denotes all hidden representations the nodes in t, after GRU encoding.

---

We employ Graph Attention Networks (GAT) with nd layers for graph learning, enabling each 24 focus other nodes that are more relevant itself. GAT also uses edge embeddings. In our model, we define edge features fn for edge Cjs as follows: (1) the edge type (parent, children, or siblings); (2) size difference 8j - 8;; (3) whether zj and x; have the same font or colour: (4) page difference; (5) position difference the differences of the coordinates of the top-left and bottom-right points of the corresponding block bounding boxes. Node Representation Update in a Subtree We transform each node-centric subtree & to a graph g = (V,E), where the nodes V {2g E t.). and the embedding of each node 2, is assigned as vy. For each node J. there are three types of edges, from its parent, from its children, and from its sib- lings. The edges between the parent/siblings and 2, may span across multiple pages, headings can be widely separated. Such edges can provide long- distance relationship information. On the other hand, the edges from children to z; provide lo- calised information about the heading. Thus, zi benefits from learning from both long-distance and local relationships.
We define three scores o/4PI d.de] and of to represent the likelihood that the node xy should be kept, deleted or moved. These scores are computed as follows:
ppl = Wkghh + bxp
del Wochs + bac
(4)
Oj
[mv]
W (wm.LPOOPPPP.. ha] Uuv
0
where POOL(PRS(h.) denotes max pooling layer on the representations of preceding siblings of zz; [...] denotes the concatenation; WAP. Wae- Wnv- bups búc. and bmv are learnable parameters. The score of Keep and Delete is inferred from the node directly, as h2 has gathered neighbourhood information with both long-distance and local rela- tionships. The score of Move is inferred from the node and its preceding siblings that the node compare itself with its preceding siblings decide whether it is sub-heading of preceding siblings. The probabilities of the node operations are com- puted with the softmax function as follows:
Pj
(5)
de)
With nodes V {a; E }. node embeddings { edge & = {egs: }. and edge embeddings fز.. the graph learning is performed as follows:
co;
co
6%
where [.] could be [kp), [de] or [mv]. The final
operation for node x; is determined as follows:
(h4 14. GAT(V,E)
9J argmax(p:)
(6)
where (ha]]ttl are the hidden representations of nodes {z,}14 in the node-centric subtree t, In practice, multiple node-centric subtrees can be merged and represented simultaneously in GPU to accelerate training and inference.
where each Si could be Keep, Delete, or Move. For each node-centric subtree t,, the model only predicts the operation ii for node and ignores other nodes. With all [0r] predicted for nodes { the original tree T will be modified shown in Algorithm 1, where node deletion is per- formed first, followed by node relocation.
4.3 Tree Modification
In this section, we discuss how the model predicts and executes modifications to the tree. We define three types of operations for each node:
We assume that all non- non-heading nodes have al- ready been deleted during the deletion step. Each node is then checked following the reading order whether it should be moved. Therefore, for a node to be moved, we can simply set its preceding sib- ling as its parent node.
1. Delete: This node predicted as not a heading and will be deleted from the tree.
2. Move: This node is predicted as a low-level heading that is a sibling of a high-level head- ing due to having the same font size in rare cases. The node 3.2.1 in Figure 2 is exam- ple. This node will be relocated to be a child as its preceding sibling as non-heading nodes have already been deleted.
The modified tree T' represents the final infer- ence output of our method, which is a ToC.
4.4 Inference and Training
For training the model, we define the ground truth label y/ of operation for each node rú, If node x; is not a heading, then its label is yi Delete. If a node x; is a heading, and there a higher-level
3. Keep: This node is predicted as a heading and does not require any operations.

---

Algorithm 1: Tree Modification
of Toc. HD evaluated using the Fl-score, which measures the effectiveness of our method in identi- fying headings from the document, which primar- ily relates to construction and modelling steps, as does not measure the hierarchical structure of ToC. For Toc, we use tree-edit-distanee similar- ity (TEDS) (Zhong et al., 2019a; Hu et al., 2022), which compares the similarity between two trees based their sizes and the tree-edit-distane (Paw- lik and Augsten, 2016) between them:
Input: A tree T, node operations {0
X de) f { E T, 9: = "delete')
= [ C T x
Reconstruct the tree T with reading order and font size.
foreach 2; T do if ÿ 'Move' then PA(x,) t -R(x;) /* Set the parent of T2 as its preceding sibling
*/
TreeEditDist(( (Tp, Ta)
end end
TEDS Tg)
max (aax(Tpl,177 (7)
Output: The modified tree T
For each document, a TEDS is computed between the predicted tree Tp and the ground-truth tree Tg. The final TEDS is the average of the TEDSs of all documents.
heading in its preceding nodes, then the label is
y Move. Otherwise, the label is y Keep. The loss is the cross entropy between 0i and yi-
Implementation Detail We use RoBERTa-base (Liu et al., 2019) as the text encoder model. We set the BFS depth = 2, and the hidden size of b, v, and h to 128. Our model is trained on NVIDIA A100 80G GPU using the Adam op- timizer (Kingma and Ba, 2015) with batch size 32. We use learning rate of le-5 for pretrained parameters, and learning rate of 1e-3 for ran- domly initialised parameters. some instances, the font size may be automatically adjusted slightly depending on the volume of text to ensure that texts that have varying fonts do not share the same font sizes. Texts with very small sizes are automatically deleted during modification.
5 Experiments
5.1 Experimental Setup
Baselines We use MTD (Hu et al., 2022) our baseline, which utilises multimodal information from images, text, and layout. The MTD consists of two steps: firstly, classifying and selecting head- ings from documents with a pre-trained language model, and secondly. modelling via GRU (Cho et al., 2014) and decoding heading relations into a hierarchical tree.
Dataset We evaluate CMM on the following ToC extraction datasets: (1) ESGDoc dataset consists of 1,093 ESG annual report documents, with 765, 1 110, and 218 in the train, development, and test sets, re- spectively. In our experiments, MTD encounters out-of-memory issues when processing some long documents in ESGDoc as it needs to model the entire document a whole. Therefore, we curated sub- dataset, denoted ESGDoc (Partial), which con- sists of documents from ESGDoc that are less than 50 pages in length. This sub-dataset contains 274, 40, and 78 documents in the train, development, and test sets, respectively. (2) HierDoc dataset (Hu et al., 2022) contains 650 scientific papers with 350, 300 in the train, and test sets, respectively. Given that the extracted text from HierDoc does not include font size, we extract font size from PDF directly using PyMuPDF.
5.2 Assumption Violation Statistics
Dataset
A1
A2
A3
Any
HierDoc
0.0
0.5
4.1
4.6
ESGDoc
0.8
1.7
8.7
10.8
Table The percentage (%) that each assumption is violated. Any denotes the percentage that the heading violates at least one assumption.
Our method is based on Assumption 1, 2, and 3. However, these assumptions do not always hold. This section presents statistics on the percentage of headings that violate these assumptions by auto- matically examining consecutive blocks along the sorted blocks { with their labels. As shown in Table 1, there are 4.6% and 10.8% of headings that contravene these assumptions in HierDoc and ESGDoc, respectively. Our current method unable
Evaluation Metrics We evaluate our method in two aspects: heading detection (HD) and the tree

---

process these non-compliant headings. Despite these limitations, our method still achieves good performance, as will be detailed in Section 5.3.
if the hierarchical information is already contained in the headings, our method may not offer many additional hierarchical insights.
5.3
Overall Results
5.4 Run-TTime Comparison
Table 2 presents the overall TEDS results on Hier- Doc and ESGDoc. Both models demonstrate good performance on HierDoc with CMM slightly outper- forming MTD. However, we observe significant performance drop 1. ESGDoc, indicating the challenge of processing complex ESG reports pared scientific papers. MTD exhibits a notably low TEDS score in ESGDoc (Full) due to the out- of-memory issue it encountered when processing certain lengthy documents. To address this, we exclude documents longer than 50 pages, result- ing in MTD achieving TEDS score of 26.9% on ESGDoc (Partial). Nevertheless, our approach CMM outperforms MTD by substantial margin. The HD Fl-score of our method outperforming MTD by 12.8% on ESGDocPartial also demonstrates the effectiveness of construction and modelling steps. Due to the violation of assumptions as discussed in Section 5.2, the improvement of our model over MTD in TOC is less pronounced compared to HD.
HierDoc
ESGDoc Partial
Model
Time
Ratio
Time
Ratio
Training
MTD
2420.6
4.6x
513.5
2.
CMM (Ours)
525.4
1.0x
241.4
1.0x
Inference
MTD
16.1
4.2x
2.7
1.3x
CMM (Ours) 3.8
1.0x
2.1
1.0x
Table 3: The GPU training and inference time in min- utes for MTD and CMM on HierDoc and ESGDoc (Partial).
Table presents the run-time comparison be- tween MTD and CMM on HierDoc and ESGDoc (Par- tial). MTD consumes 4.6x and 1x more time for training and 4.2x and 1.3x more time for inference HierDoc and ESGDoc, respectively. Different from MTD, CMM does not need model all possible pairs of headings. Instead, it only predicts whether a node should be deleted or relocated, thereby re- ducing the computational time.
HierDoc
ESGDoc F.
ESGDoc P.
Model
HD
ToC
HD
ToC
HD
ToC
Compared to MTD, our model exhibits higher efficiency on HierDoc compared to ESGDoc. This could be attributed to the larger number of edges in the graphs constructed from node-centric subtrees in our method for ESGDoc. ESG annual reports of- ten contain numerous small text blocks, such as "$5,300m", " "14,000", and "3,947 jobs", as illus- trated in the first example of Figure 1. Our method treats these individual texts as separate nodes in both the trees and graphs, leading to significant increase in the number of edges in ESGDoc com- pared to HierDoc.
MTD
96.1
87.2
12.7
12.8
40.4
26.9
CMM (Ours)
97.0
88.1
55.6
33.2
53.2
30.0
Table 2: Hcading detection (HD) in Fi-score and ToC in TEDS (%) of MTD and CMM HierDoc, ESGDocFull (F), and ESGDocPartial (P.).
Our model's performance on ESGDoc (Full) demonstrates its scalability in handling ESG re- ports with diverse structures and significantly lengthy text. The comparable TEDS scores be- tween CMM and MTD on HierDoc can be potentially attributed the nature of scientific papers. For instance, headings in scientific papers such as "5 Experiments" and "5.1 Experimental Setup", pro- vide explicit indication of their hierarchical rela-
5.5 Ablation Study
Table 4 illustrates how different components in CMM contribute to performance:
tionships within the headings themselves. The pres- ence of section numbering such as "5" and "5.1" makes it casier to determine their hierarchical level such as the latter being a sub-heading of the former. Our method introduces hierarchical information via the reading order and font sizes, and learns tree node representations by simultancously consider- ing long-distin distance and local relationships. However,
w/ page-based division CMM divides the docu- ment into subtrees based on the tree structure. We substitute the tree-based division with a page-based one. Initially, the document is divided using a window of pages with 2-page overlap. All other steps remain unchanged, including the mod- elling node-centric subtrees. The choice of the

---

HierDoc
ESGDoc
HierDoc
ESGDoC
Model
88.5
HD
ToC
HD
ToC
CMM
97.0
88.1
55.6
33,2
88.0
w/ page division
96.7
87.8
52.3
30.1
31.0
w/o GRU
96.8
87.7
50.8
30.5
87.5
30.0
w/o GNN
96.4
87.4
44.0
24.9
1
3
Figure 3: Study of BFS depth na of CMM on HierDoc and ESGDoc in TEDS.
Ablation Study of CMM on HierDoc and ESGDoc with heading detection (HD) and ToC results reported in Fl-score and TEDS (%), respectively.
Table 4:
secondary headings, and main body content in a three-tier configuration. (2) The constructed initial tree inherently positions related headings in close proximity according to their semantic relationships, without regard for their relative page placement. As consequence, a hcading's most relevant con- textual information predominantly emerges from its immediate neighbours within the tree. For ex- ample, when examining heading 1.2., information from heading 1. (nd = 1) offers a comprehensive overview of the encompassing chapter. Simultane- ously, heading 2. (nu 2) can provide supplemen- tary insights, such affirming that heading 1.2. is nested within the domain of heading 1., rather than heading 2. However, delving into deeper levels may become redundant. For example, heading like 2.2. (na 3), situated more distantly in the semantic space, would not notably enhance the un- derstanding of heading 1.1.
page number for division is made to keep a simi- lar GPU memory consumption. This results in performance drop of 0.3% on HierDoc and 3.1% ESGDoc. The page-based division impedes long- distance interaction, resulting in lack of connec- tion between high-level headings.
w/o GRU We exclude the GRU in Eq. (2) and di- rectly set v = b, The results show a performance drop of 0.4% on HierDoc and 2. 2.7% on ESGDoc.
w/o GNN We exclude the GNN in Eq. (3) and directly set h4 = uy. This results in more signif- icant performance drop of 0.7% on HierDoc and 8.3% on ESGDoc. With GNN, each heading can gather information from other long-distance and local body nodes effectively and simultaneously. As shown in Table 4, there a larger perfor- mance drop on ESGDoC compared to HierDoc. This can be attributed to the same reason outlined in Sec- tion 5.3: inferring hierarchical relationships from headings themselves is easier in HierDoc than in ESGDoc. Therefore, the removal of components that introduce hierarchical relationships does not significantly harm the performance on HierDoc. Figure 3 demonstrates how performance varies with different values of na, the depth of neighbour- hood during BFS for constructing node-centric sub- trees. Due the nature of the data, there is limited improvement observed on HierDoc as na increases. Notably, there is a substantial increase TEDS from 14 to n = 2, but the improvement be- comes negligible when ny > 2. Therefore, we select ny = considering the trade-off between performance and efficiency.
Some case studies illustrating the ToC extrac- tion results of CMM on ESGDoc are presented in Ap- pendix C.
Conclusion and Future Work
In this paper, have constructed a new dataset, ESGDoc, and proposed novel framework, CMM, for table of contents extraction. Our pipeline, con- sisting of tree construction, node-centric subtree modelling, and tree modification stages, effectively addresses the challenges posed by the diverse struc- tures and lengthy nature of documents in ESGDoc. The methodology of representing document initial full tree, and subsequently predicting node operations for modification, and further lever- aging the tree structure for document segmentation, can provide valuable insights for other document analysis tasks.
The primary factors contributing the negligi- ble improvement n4 > 2 may include: (1) A signifi- cant portion of documents exhibit a linear structure. To illustrate, when n, = 2, it corresponds to hier- archical arrangement featuring primary headings,

---

Acknowledgements
References
Christopher M. Bishop and Nasser M. Nasrabadi. 2006. Pattern recognition and machine learning. J. Elec- tronic Imaging, 16:049901.
This work was funded by the the UK Engineering and Physical Sciences Research Council (grant no. EP/0017112/1, EP/T017112/2 EP/VO48597/1). YH is supported by a Turing Al Fellowship funded by the UK Research and Innovation (grant no. EP/V020579/1, EP/V020579/2).
L. Breiman. 2001. Random forests. Machine Learning. 45:5-32.
Shaked Brody, Uri Alon, and Eran Yahav. 2021. How attentive are graph attention networks? ArXiv, 144911
Limitations
Kyunghyun Cho, Bart van Merriënboer, Caglar Gul- cehre. Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods Natu- rul Language Processing, pages 17341733, Doha, Qatar. Association for Computational Linguisties.
Our method exhibits two primary limitations. Firstly, it relies on the extraction of font size. For documents in photographed or scanned forms, an additional step is required to obtain the font size before applying our method. However, with the prevailing trend of storing documents in electronic formats, this limitation is expected to diminish in significance.
Antoine Doucet, Gabriella Kazai, Bodin Dresevic, Alek- sandar Uzelac, Bogdan Radakovic, and Nikola Todic. 2011. Setting up competition framework for the evaluation of structure extraction from ocr-ed books. International Journal on Document Analysis and Recognition, 14:45-52.
Secondly, our method grounded Assump- tions 1, 2, and 3. As discussed in Section 5.2, our current method encounters difficulties in scenarios where these assumptions are not met. Some ex- amples of such assumption violations are provided in Appendix B. However, it is worth noting that these assumption violations primarily impact the modification step in our cnnstructionmmddlllinn modification approach. If we focus solely on the construction and modelling steps, our method still outperforms MTD in heading detection. There- fore, future efforts to enhance the modification step, which is susceptible to assumption violations, could hold promise for improving the overall per- formance of our approach.
Zhangxuan Gu, Changhua Meng, Ke Wang, Jun Lan, Weiqiang Wang, Ming Gu, and Liqing Zhang. 2022. Xylayoutim: Towards layout-aware multimodal net- works for visually-rich document understanding. 2022 IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 3573-4582.
Jackyu Ha, Robert Haralick, and Thsin T Phillips. 1995. Recursive cut using bounding boxes of connected components. In Proceedings ternational Conference on Document Analysis and Recognition, volume 2, pages 952-955. IEEE
3rd In-
Pengfei Hu, Zhenrong Zhang, Jianshu Zhang, Jun Du, and Jiajia Wu. 2022. Multimodal tree decoder for ta- ble of contents extraction in document images. 2022 26th International Conference on Pattern Recogni- tion, pages 1756-1762.
Ethics Statement
The ESG annual reports in ESGDoc are indepen- dently published by the companies and are publicly accessible. DisponsibilityRepott.ooo com compiles these ESG annual reports, which are also accessi- ble directly on the respective companies' websites. There are also other websites such as CSRWIRE? and ciwaaiiiabiiityrrootts.ooom
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd Interna- tional Conference on Learning Representations.
Chen-Yu Lee, Chun-Liang Li, Timothy Dozat, Vincent Perot, Guolong Su. Nan Hua, Joshua Ainslie, Ren- shen Wang. Yasuhisa Fujii, and Tomas Pfister. 2022. FormNet: Structural encoding beyond sequential modeling form document information extraction. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume
serve as reposito- ries for these reports. Because these reports are publicly available, the use of such data for research purpose is not anticipated to present any ethical concerns.
1: Long Papers), pages 3735-3754, Dublin, Ireland. Association for Computational Linguistics.
"https: htttt::://ww.rrsssssiiilli. responsibilityrepoon //ttps://www.. csrwire commreppots 'https: http:///wssstaaabliiyryrreeeeeee
Chen-Yi Yu Lee, Chun-Liang Li, Hao Zhang, Timothy Dozat, Vincent Perot, Guolong Su, Xiang Zhang. Kihyuk Sohn, Nikolay Glushney, Renshen Wang.

---

Joshua Ainslie, Shangbang Long, Siyang Qin, Ya- suhisa Fujii, Nan Hua, and Tomas Pfister. 2023. 2ormNetV2: Multimodal graph contrastive learn- ing for form document information extraction. In Proceedings of the 61st Annual Meeting of the As- sociation for Computational Linguistics ( (Volume I: Long Papers), pages 901 1-9026, Toronto, Canada. Association for Computational Linguistics.
Zejiang Shen. Kyle Lo, Lucy Lu Wang. Bailey Kuehl, Daniel S. Weld, and Doug Downey, 2022. VILA: Im- proving structured content extraction from scientific PDFs using visual layout groups. Transactions of the Association for Computational Linguisties, 10:376- 392.
Carlos Soto and Shinjae Yoo. 2019. Visual detection with context for document layout analysis. In Pro- ceedings of the 2019 Conference on Empirical Meth- ods in Natural Longuage Processing and the 9th In- ternational Joint Conference on Natural Language Processing, pages 3464-3470, Hong Kong, China. Association for Computational Linguisties.
Chenxia Li, Ruoyu Guo, Jun Zhou, Mengtao An, Yun- ing Du, Lingfeng Zhu, Yi Liu, Xiaoguang Hu, and Dianhai Yu. 2022. Pp-structurev2: A stronger docu- ment analysis system. ArXiv, abs/2210.05391.
Minghao Li, Yiheng Xu, Lei Cui, Shaohan Huang, Furu Wei, Zhoujun Li, and Ming Zhou. 2020. DocBank: benchmark dataset for document layout analy- sis. In Proceedings of the 28th International Confer- ence Computational Linguistics, pages 949-960, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Suppawong Tuarob, Prasenjit Mitra, and C. Lee Giles. 2015. A hybrid approach to discover semantic hier- archical sections in scholarly documents. 2015 13th International Conference on Document Analysis and Recognition, pages 1081-1085.
Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio', and Yoshua Ben- gio. 2017. Graph attention networks. ArXiv, abs/1710.10003.
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019.
Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprini arXiv: 1907.1 I 1692.
Yang Xu, Yiheng Xu, Tengchao Lv, Lei Cui, Furu Wei, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang. Wanxiang Che, Min Zhang, and Lidong Zhou. multi-modal pre-training for visually-rich document understanding. In Proceed- ings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 2579-2591, Online. Association for Computational Linguisties.
Abhijith Athreya Mysore Gopinath, Shomir Wilson, and Norman Sadeh. 2018. Supervised and unsupervised methods for robust separation of section titles and prose text in web documents. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 850-855, Brussels, Bel- gium. Association for Computational Linguistics.
2021a. Lay:
Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, and Ming Zhou. 2019, Layoutim: Pre-training of text and layout for document image understanding. Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining.
Anoop M. Namboodiri and Anil K. Jain. 2007. Doc- ument Structure and Layout Analysis, pages 29 48. Springer London, London.
Mateusz Pawlik and Nikolaus Augsten. 2016. Tree edit distance: Robust and memory-efficiett. Information Systems, 56:157-173
Yiheng Xu, Tengchao Lv, Lei Cui, Guoxin Wang, Yi- juan Lu, Dinei A. F. Florêncio, Cha Zhang. and Furu Wei. I0. Layoutxlm: Multimodal pre-training for multilingual visually-rich document understanding. ArXiv, abs/2 104.08836.
Qiming Peng, Yinxu Pan, Wenjin Wang, Bin Luo, Zhenyu Zhang, Zhengjie Huang, Yuhui Cao, Wei- chong Yin. Yongfeng Chen, Yin Zhang, Shikun Feng, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. 2022. ERNII-layout: Layout knowledge enhanced pre- training for visually-rich document understanding. Findings of the Association for Computational Linguistics: EMNLP 2022, pages 3744-3756, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguisties.
Peng Zhang, Can Li, Liang Qiao, Zhanzhan Cheng, Shiliang Pu, Yi Niu, and Fei Wu. 2021. Vsr: A unified framework for document layout analysis combining vision, semantics and relations. ArXiv, abs/2105.0622 06220..
Xu Zhong, Elaheh Shafieibavani, and Autonio Jimeno- Yepes. 2019a. Image-based table recognition: Data, model, and evaluation. ArXiv, abs/1911.
Birgit Pfitzmann, Christoph Auer, Michele Dolfi, Ahmed S. Nassar, and Peter Staar. 2022. Doclaynet: A large Human--nnooatedd dataset for document- layout segmentation. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discov- ery and Data Mining, KDD '22, page 3743-3751, New York, NY, USA. Association for Computing Machinery.
Xu Zhong, Jianbin Tang, and Antonio Jimeno- Yepes. 2019b. Publaynet: Largest dataset ever for document layout analysis. 2019 International Conference on Document Analysis and Recognition, pages 1015- 1022.

---

Appendix
economic impacts" and "Transmission System In- vestments" are headings situated at the same hi- erarchical level but have distinct font sizes. This dissimilarity could potentially lead to confusion for human readers, questioning whether these two headings should be placed within the same hierar- chical level. In the second example, "Suutainabiliity Fund purchases" and "Spend by solution type" are also headings the same level, with subtly differ- font sizes, 11 and 10, respectively. While this difference may go unnoticed by humans, it does impact the performance of our method.
A Algorithm of Building an Initial Tree
Algorithm 2 describes how to build an initial full tree from a document.
Algorithm 2: Building a Tree based on Reading Order and Font Sizes
Input: Root node all blocks {-i with their corresponding sizes {8; 188
for i 1 to do
j < while >= 0 do
C Case Study
if', j = 0 then
PA(a)) - r /* Set r as the
Figure A3 and Figure A4 illustrate a favourable scenario and unfavorable one for CMM within the context of ESGDoc. The favourable case in Fig- ure A3 demonstrates the capability of our model to handle lengthy document, where it generates a high quality tree structure.
parent node of 2; */
break
else if 8j > 5; then
PA(x;) - Fj /* Set zj as
the parent node of ii
*/
break
Conversely, Figure A4 represents a challeng- ing scenario where our model encounters diffi culties across multiple nodes. Figure A5 further elaborated on this issue, showcasing four example pages of the unfavorable case illustrated in Fig- ure A4. the top-left page, CMM incorrectly re- tain the non-heading "ABOUT BRANDYWINES This is a challenging case as "ABOUT BRANDY- WINE" is prominently displayed in large font at the top-left corner of the page, making it diffi- cult to identify ation occurs the top-right page, where CMM in- correctly keeps the non-heading "ENVIRONMEN- TAL PROGRESS". In this instance, "ENVIRON- MENTAL PROGRESS" enlarged to emphasise the company's s achievements.
end
j
end
end
Output: tree T with root node r.
B Assumption Violation Examples
For Assumption 1, upon manually inspecting some samples, we found that errors in these particular cases were linked to the errors in the X--cut al- gorithm (Ha al., 1995), resulting an incorrect arrangement of text blocks.
non-heading. A similar situ-
Figure A1 presents two examples where As- sumption 2 is not satisfied. In the first example, the term "The way work" serve as the parent- heading of "Corporate governance" which is sub- heading, but featuring smaller font size. Despite the smaller font size of "The way we work", is clearly delineated from the sub-headings below by two green horizontal lines. However, our method focuses solely on text, neglecting visual cues such as these lines. In the second example, "COMMU- NITY OUTREACH" is sub-heading under "SO- CIAL RESPONSIBILITY"" but is has larger font size, this page emphasises community achieve- ments.
For the bottom two pages in Figure A5, CMM might encounter confusion between headings with coloured lead-in sentences. In the bottom-left page, "MANAGING CLIMATE RISK" functions as a heading and follows a pattern similar to other lead-in sentences such as "GOVERNANCE" and "STRATEGY AND RISK MANAGEMENT". They typically begin with a large, colored sentence fol- lowed by a paragraph. The bottom-right page presents similar challenge. "OUR TENANTS" and "VALUED PARTNERSHIPS" share similar pattern, with the former being a heading, whereas the latter not. The determination of "MANAGING CLIMATE RISK" and "OUR EMPLOYEES" as headings is primarily based on their position and
Figure A2 presents two instances where Assump- tion 3 is violated. In the first example, 'indirect

---

SECWI RERNNDEALT IT
OUR ASSOCIATES
COVID-19 SAFETY IN STORES
15, 15,000
50%
COMVUNITY OUTREACE
50+
84, 4,0000.mm
10,000
Figure A1: Two examples in ESGDoc where Assumption 2 is violated, one in portrait and the other in landscape orientation.
Busae by Purr pareoeesss
quet fect sun recotrccaae bust Co0r
clear desige of seed aur tiiee
propp
24%
23%
15%
9%
9%
8%
6%
3%
3%
Figure A2: Two examples in ESGDoc where Assumption 3 is violated, one in portrait and the other in landscape orientation.
colour. However, it is worth noting that CMM does not use visual information, making it difficult for the model to handle such scenarios. Future work
could explore the integration of visual information to enhance the model's performance in handling these situations.

---

Bros BROWASS
Abdt
Formort fon the ChafConzie Cfforr
Poblo
Sata Filllty sprian Sayyes
Gold Ea porrs Govarance
Procenmert
Protect humen applychie
wwloreerellaangeeeee liding siket
Henggra woupree antireeiiose givenno
Trnnoutt ant
carusn sraniyy
oppartiity
Eypsyee
Thaiking and preketioon Semiipinntt
Buille suolaan digntivv
BB Conrent wiie
Piblo sayees
Eanp ineoeed
Va aniiciill cervies
wagestgghaioooo ligtt tuplyy
Wits sad addee
zndiagidie
Ceriubeee boilt saliity
Stateneet
worit
Figure A3: A favourable case for CMM on ESGDoc. Blocks highlighted in red represent incorrect predictions.

---

Groundouthh ToC
SGAS/AS524 SOCOUMTIOO
KANGG CURVATCFRE
SFRCENCY
.0155
150 GOVEENAACEE
EXCOUTVCC
Prediced
AMEAAGI RON
ARCUTI SCOTTAAETTON PR00
QuAuIty
AFTONCICSS EA
Figure A4: unfavorable case for CMM on ESGDoc. Blocks highlighted in red represent incorrect predictions.
EMTRCIEENTAL PHOGRESS
BRANDWAINE
51.7%
1R1
34.5A
S
3/11
SAS CLIMATE
Drasel
Figure A5: Four example pages of the unfavorable highlighted in Figure A4, CMM preserves non-headings in the top two pages, while deleting headings in the bottom two pages.


Reviews:
[{"review_id": "OUYNOr7aCo", "reviewer": "Reviewer_Z2mu", "summary": "", "questions": "Have you considered releasing the codebase and dataset for your approach?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Q1**: A comparison with other state-of-the-art models would provide valuable insights and further strengthen the contribution of the proposed framework.\n\n**A1**: The task we've undertaken is relatively novel and the only existing approach applied on the task that we are aware of is MTD. Although we did explore other potential baselines such as chatPDF as well as certain specialised plugins of chatGPT and GPT4, these methodologies have not been made available to the public. Furthermore, both chatGPT and GPT4 inherently come with a constraint on input  length, which prevents them from effectively processing extensive documents all at once. Due to these limitations, we made the decision to not consider them as baselines in our study. \n\n--- \n> **Q2**: Have you considered releasing the codebase and dataset for your approach?\n\n**A2**: Yes, we will release them as mentioned in the footnote of Line 32.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper demonstrates a strong baseline dataset comparison, an ablation study and a solid methodology foundation to address the problem effectively. The experimental design and results are presented clearly and concisely. Additionally, the detailed elaboration on the dataset construction enhances the reproducibility of the research.", "reasons_to_reject": "A comparison with other state-of-the-art models would provide valuable insights and further strengthen the contribution of the proposed framework.", "questions_to_authors": "Have you considered releasing the codebase and dataset for your approach?", "paper_topic_and_main_contributions": "This paper presents a scalable framework for Table of Contents (ToC) extraction from ESG reports. The main contributions include the introduction of a new dataset, ESGDoc, a novel framework for processing documents in a construction-modelling-modification manner, and a graph-based method for document segmentation and modelling.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "-", "Typos_Grammar_Style_and_Presentation_Improvements": "Please ensure consistent use of abbreviations throughout the paper. For example, use \"ToC\" (001) instead of \"Toc\"(002) for Table of Contents (ToC) in all sections of the manuscript."}, "extra_text": {}}, {"review_id": "4Quh7Uw17p", "reviewer": "Reviewer_eEPP", "summary": "", "questions": "Question A: Have the authors checked if instead of decomposing the PDF into blocks and then constructing the tree, instead convert the PDF into say HTML and use the DOM tree? Would this avoid the initial tree construction in Algorithm 2? \n\nQuestion B: Figure A3 shows ground truth ToC tree of ESGDoc. What are the criteria used for annotating {Keep, Move, Delete} other than those described in \\subsection {Tree Modification}?\n\nQuestion C: Do you have insights on why the TEDS performance jumps in Figure 3 when d goes from 1 to 2 for n_d and shows only a minute increase afterwards?\n\n\n", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Q1**: Have the authors checked if instead of decomposing the PDF into blocks and then constructing the tree, instead convert the PDF into say HTML and use the DOM tree? Would this avoid the initial tree construction in Algorithm 2?\n\n**A1**: The distinction between the DOM tree and our tree construction methodology holds significant importance. The DOM tree primarily focuses on the layout details and the storage of the PDF file. On the other hand, our constructed tree prioritises semantic content organisation, aligning with human cognitive patterns.\n\nLet’s consider an example: Heading 2., as a first-level heading, shares a stronger semantic relation with other first-level headings, such as Heading 5. However, its relation to a second-level heading, say 4.2, is weaker, despite the possibility of Heading 4.2 being physically closer to Heading 2 in terms of page sequence. The extent of such correlations can be adjusted by modifying the parameter n_d. Unfortunately, the DOM tree falls short in capturing these nuanced semantic relationships.\n\n\n--- \n> **Q2**: Figure A3 shows ground truth ToC tree of ESGDoc. What are the criteria used for annotating {Keep, Move, Delete} other than those described in \\subsection {Tree Modification}?\n\n**A2**: To begin, we annotate all headings as “keep” and non-headings as “delete”. Then for each heading, we check if its ground-truth parent heading corresponds to one of its preceding siblings in the original tree. If yes, we change its annotation from “keep” to “move”.\n\n\n--- \n> **Q3**: Do you have insights on why the TEDS performance jumps in Figure 3 when d goes from 1 to 2 for n_d and shows only a minute increase afterwards?\n\n**A3**: The primary factors contributing to this phenomenon may include: \n(1) A significant portion of documents exhibit a linear structure. To illustrate,  when n_d=2, it corresponds to a hierarchical arrangement featuring primary headings, secondary headings, and main body  content in a three-tier configuration. \n(2) The constructed initial tree inherently positions related headings in close proximity according to their semantic relationships, without regard for their relative page placement. As a consequence, a heading's most relevant contextual information predominantly emerges from its immediate neighbours within the tree. \n\nFor example, when examining heading 1.2., information from heading 1. (n_d=1) offers a comprehensive overview of the encompassing chapter. Simultaneously, heading 2. (n_d=2) can provide supplementary insights, such as affirming that heading 1.2. is nested within the domain of heading 1., rather than heading 2. However, delving into deeper levels may become redundant. For example, a heading like 2.2. (n_d=3), situated more distantly in the semantic space, would not notably  enhance the understanding of heading 1.1.\n\n\n--- \n> **Q4**: It would be helpful to add the inference time as well in addition to the existing training time since in the paper run-time is the term used instead of training time.\n\n**A4**: The inference time in terms of the number of minutes is as follows:\n|  | HierDoc | ESGDoc P. |\n|---|---|---|\n| MTD | 16.1 |  2.7 |\n| CMM | 3.8 | 2.1 |\n| Ratio | 4.2x | 1.3x |\nWe will add it to the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The authors have introduced the idea of splitting the original PDF tree into sub-trees that will allow to parse large PDFs without facing out of memory errors.\n\nThe authors have modified their custom dataset that would otherwise have not been able to handle by the baseline approach during evaluation.\n\nContribution towards a new dataset that focuses on a more complex and longer document size than the existing one.\n\nA significant improvement in training time and performance over the baseline. \n", "reasons_to_reject": "None.", "questions_to_authors": "Question A: Have the authors checked if instead of decomposing the PDF into blocks and then constructing the tree, instead convert the PDF into say HTML and use the DOM tree? Would this avoid the initial tree construction in Algorithm 2? \n\nQuestion B: Figure A3 shows ground truth ToC tree of ESGDoc. What are the criteria used for annotating {Keep, Move, Delete} other than those described in \\subsection {Tree Modification}?\n\nQuestion C: Do you have insights on why the TEDS performance jumps in Figure 3 when d goes from 1 to 2 for n_d and shows only a minute increase afterwards?\n\n\n", "paper_topic_and_main_contributions": "The paper talks about the challenges faced by the existing system in extracting Table of Contents (ToC) from long and complex PDF documents and offers a new scalable ToC extraction method. The proposed approach consists of constructing a tree structure of the extracted blocks of text from the PDF, parsing through sub-trees and capturing each node's contextual information and finally structuring a new tree with the original nodes being either kept as is, deleted or moved. This work is supplemented by introducing a new dataset called ESGDoc that consists of 1093 ESG (Environmental, Social, and Governance) annual reports from 563 companies. The paper shows how their approach improves over the baseline on the ESGDoc as well as on the existing HierDoc datasets in terms of both accuracy as well as the ability to handle longer length documents.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "In line 615 instead of n_{d} > 3 it can be n_{d} \\geq 3 due to only a minute improvement witnessed in Figure 3.\n\nIt would be helpful to add the inference time as well in addition to the existing training time since in the paper run-time is the term used instead of training time."}}, {"review_id": "dAbj8eLwjS", "reviewer": "Reviewer_Eyfj", "summary": "", "questions": "A. Your articulation of assumptions is very helpful, but your Assumption\n3 seems a little problematic, based on Table 1.  To what extent does\nthe data's violation of this assumption depress your performance\nscores? \n\nB. Your statement that MTD \"only uses\" GRUs to capture candidate header\ncontext is slightly confusing.  Did you mean to say that MTD only\nmodels local context, or does the use of GRUs somehow contribute to\nthe limitation of MTD that you're trying to articulate?  More\ngenerally, I think you need to say more about MTD, since it's your\nonly point of comparison with prior work.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Q1**: The motivation for the work is underdeveloped. In the introduction, the paper merely notes that this task is a focus of prior work, but it was unclear to me what solving this problem enables. Certainly, the research is of academic interest, and perhaps one can imagine a document visualizer or summarization system based on this work, but what else might it lead to? This question is all the more acute for the ESG domain, where accuracy is low and subject matter varies widely. In the scientific domain, because of strong structural presentations, one can imagine basing downstream automation off the interpretation of a paper, but is this possible in the contributed domain? What could you create from this functionality, other than a kind of automated document thumbnail generator?\n\n**A1**: The motivation behind the work is rooted in the observation that while traditional summarisation offers a concise representation of a document’s content, a Table of Contents (ToC) presents a structured and hierarchical summary. This structural organisation in a ToC provides a comprehensive pathway for pinpointing specific information. For example, when seeking information about a company's carbon dioxide emissions, a ToC enables a systematic navigation through the information hierarchy, such as environment -> climate -> greenhouse gases -> carbon dioxide. In contrast, conventional summarisation might only provide a vague indication of such information, requiring sifting through the entire document for precise detail. \n\nWhile our primary focus in this paper centres on the ESG domain, the automated generation of a ToC holds relevance for lengthy documents across various domains. Furthermore, the construction and modelling components introduced in our methodology hold potential for future applications in other document-related tasks.\n\n\n--- \n> **Q2**: The experiments are not particularly extensive, employing one baseline and two problem sets. The baseline is not described in sufficient detail for the reader to understand the tradeoffs implied by the comparison of CMM and MTD. The experiments seem to imply that CMM is simply to be preferred in all cases, but there's not enough here for a reader to reach that conclusion confidently.\n\n**A2**: We will add more detail about MTD in the paper. Our choice to limit the baseline to MTD is attributed to MTD proposing this task, and the limited existing work on this particular task. \n\nCMM holds a comparative advantage over MTD for two main reasons:\n\n(1) *Incorporation of auxiliary information in the modelling step*. CMM leverages size and three assumptions as additional cues to construct an initial tree, already imparting some hierarchical context. In contrast, MTD starts the modelling process from scratch, without leveraging such external cues.\n\n(2) *Task simplification in the modification step*. We simplify a task of paired heading classification in MTD into singular heading classification, resulting in a reduction of decoding time complexity from O(n^2) to O(n). However, such simplification suffers from the issue of assumption violation (see Section 5.2, Section Limitations, and Appendix C). The modification step, only containing three operations, does not completely fix the problem of assumption violation, which in turn limits the performance improvement in the modification step. To illustrate,  even with a flawless modification step, the ToC scores are 95.4% on HierDoc and 84.4% on ESGDoc. Nonetheless, in this paper, our primary focus is on presenting a feasible solution for dealing with lengthy documents with complex visual layouts and our model exhibits superior performance compared to MTD.\n\n\n--- \n> **Q3**: The paper doesn't provide much in the way of insights into the ESGDoc task. It's evident that the problem is much harder than HierDoc, but why? I found myself wondering whether it's simply underspecified and somewhat arbitrary what TOC emerges from a typical document in this domain. More concretely, how would humans do in performing this task? If humans can reliably reconstruct the given TOC, then I think we have the makings of an interesting challenge. The paper doesn't offer much of an error analysis to provide insights into why, for example, CMMs ability to identify ESG TOC entries hovers around 50%. Diversity of presentation conventions? Document length? Something else that might be address with better understanding of section contents? The paper doesn't provide satisfactory answers to these questions.\n\n**A3**: HierDoc's relatively straightforward nature can be attributed to the structured formatting found in  scientific papers, where heading numbering such as \"1. Introduction\" facilitate easier classification. This consistent structuring and numbering mechanism offers clues that models can easily rely on. (see Line 053-057, Line 539-546). On the other hand, ESGDoc, in contrast, exhibits significant layout variability. The content structure, the wording of headings, and their hierarchical ordering can differ widely, depending on the reporting company's chosen narrative (see Fig 1, A1, A2, and A5 as examples).\n\nDue to the page limitation, case study and error analysis are shown in Appendix C and D. The ToC of each ESG report depends on the style of the company and the narrative they aim to present. For instance, the phrase \"About this report\" can manifest in multiple ways across reports. In some instances, it might stand as an independent heading, while in others, it might be incorporated within the body.  This diversity can be perplexing, not only for models but even for human readers. More details are discussed in Appendix C and D.\n\nThe ESGDoc dataset encompasses reports varying from 4 pages to 521 pages, with an average of 72 pages, whereas HierDoc ranges from 2 pages to 298 pages, with an average of 19 pages. (See Section 3). On average, ESGDoc is 3.8 times longer than HierDoc. Further elaboration on this aspect will be incorporated in the revised version. \n\n\n--- \n> **Q4**: Your articulation of assumptions is very helpful, but your Assumption 3 seems a little problematic, based on Table 1. To what extent does the data's violation of this assumption depress your performance scores?\n\n**A4**: We did not conduct such experiments to avoid impacting the original dataset distribution.  Modifying or removing data that doesn't conform to assumptions could result in altered data distributions that deviate from real-world data. Nevertheless, we can offer some insights into this matter. Assumption violation mainly affects the modification phase, as the modification with only three operations cannot address all assumption violation problems. \n\nComparing CMM with MTD, CMM outperforms MTD in heading detection (HD) by 13.6% but only by 4.2% in ToC, as shown in Table 2. HD assessment only evaluates the construction and modelling steps, while ToC evaluation encompasses all three steps. The limited efficacy of the modification step in addressing assumption violations leads to a low ToC score. Moreover, as mentioned in A2, assumption violation results in a performance limit for the modification step, with ToC score of 95.4% for HierDoc and 84.4% for ESGDoc.\n\n\n--- \n> **Q5**: Your statement that MTD \"only uses\" GRUs to capture candidate header context is slightly confusing. Did you mean to say that MTD only models local context, or does the use of GRUs somehow contribute to the limitation of MTD that you're trying to articulate? More generally, I think you need to say more about MTD, since it's your only point of comparison with prior work.\n\n**A5**: MTD initially uses BERT to derive text representations for various segments of a document, including both the headings and bodies. Then, GRU is employed to model these representations, forming a sequence of vectors representing headings and bodies. The phrase \"only uses\" emphasises that MTD relies solely on the GRU to capture contextual connections among candidate headings from scratch, without leveraging hierarchical cues as mentioned in A2. While GRUs are adept at capturing sequential dependencies, their efficiency diminishes when dealing with extremely long sequences, as encountered in documents that comprise thousands of textual units, leading to potential gaps in comprehending context. \n\n\n--- \n> **Q6**: You might want to say explicitly that you're releasing ESGDoc to support further research, rather than just \"introducing\" the dataset.\n\n**A6**: Yes, we will release them as mentioned in the footnote of Line 32 in the paper.\n\n\n--- \n> **Q7**: You talk about document understanding a fair amount in the front matter, but never define it. A succinct statement concerning what you think research into document understanding is about (and what uses it might have) would help make the scope of your research more concrete.\n\n**A7**: Document understanding refers to interpreting and comprehending the structure, content, and semantics of documents. Previous methods of document understanding might prioritise layout details, such as classifying titles, figures, tables, and body text. Conversely, in this paper, we focus more on semantic comprehension, which encompasses not just recognising individual words or sentences but also grasping the hierarchical relationships and context within the document. It's akin to how humans process lengthy reads: we don't just understand isolated facts, but also how those facts interrelate and the overarching narratives they collectively form.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper enriches prior work on automated TOC creation by\ncontributing a new domain that is considerably more difficult than\npreviously used problem sets.  Where accuracies on HierDoc hover\naround 90%, raising concerns of saturation, performance on ESGDoc is\nmuch lower, suggesting that it may be an interesting resource for\nresearch into document understanding.\n\nThe approach proposed in the paper (CMM) is apparently effective.  The\npaper musters convincing evidence that CMM should be preferred to MTD,\nthe SOTA model used as baseline in the paper.  Improvements over MTD\nin both efficiency and accuracy are consistent and, in the case of\nESGDoc, substantial.\n\nThe algorithm is intuitive and its explication solid.  It should be\nstraightforward for a knowledgeable reader to implement the approach\nand, with the data contributed by the paper, replicate the results.", "reasons_to_reject": "The motivation for the work is underdeveloped.  In the introduction,\nthe paper merely notes that this task is a focus of prior work, but it\nwas unclear to me what solving this problem enables.  Certainly, the\nresearch is of academic interest, and perhaps one can imagine a\ndocument visualizer or summarization system based on this work, but\nwhat else might it lead to?  This question is all the more acute for\nthe ESG domain, where accuracy is low and subject matter varies\nwidely.  In the scientific domain, because of strong structural\npresentations, one can imagine basing downstream automation off the\ninterpretation of a paper, but is this possible in the contributed\ndomain?  What could you create from this functionality, other than a\nkind of automated document thumbnail generator?\n\nThe experiments are not particularly extensive, employing one baseline\nand two problem sets.  The baseline is not described in sufficient\ndetail for the reader to understand the tradeoffs implied by the\ncomparison of CMM and MTD.  The experiments seem to imply that CMM is\nsimply to be preferred in all cases, but there's not enough here for a\nreader to reach that conclusion confidently.\n\nThe paper doesn't provide much in the way of insights into the ESGDoc\ntask.  It's evident that the problem is much harder than HierDoc, but\nwhy?  I found myself wondering whether it's simply underspecified and\nsomewhat arbitrary what TOC emerges from a typical document in this\ndomain.  More concretely, how would humans do in performing this task?\nIf humans can reliably reconstruct the given TOC, then I think we have\nthe makings of an interesting challenge.  The paper doesn't offer much\nof an error analysis to provide insights into why, for example, CMMs\nability to identify ESG TOC entries hovers around 50%.  Diversity of\npresentation conventions?  Document length?  Something else that might\nbe address with better understanding of section contents?  The paper\ndoesn't provide satisfactory answers to these questions.", "questions_to_authors": "A. Your articulation of assumptions is very helpful, but your Assumption\n3 seems a little problematic, based on Table 1.  To what extent does\nthe data's violation of this assumption depress your performance\nscores? \n\nB. Your statement that MTD \"only uses\" GRUs to capture candidate header\ncontext is slightly confusing.  Did you mean to say that MTD only\nmodels local context, or does the use of GRUs somehow contribute to\nthe limitation of MTD that you're trying to articulate?  More\ngenerally, I think you need to say more about MTD, since it's your\nonly point of comparison with prior work.", "paper_topic_and_main_contributions": "The paper addresses the automatic generation of tables of contents\nfrom PDF documents, extending previous work in the area, which has\nfocused heavily on scientific papers, with a corpus of corporate\nannual reports (ESGDoc).  Compared with scientific documents,\ndocuments in ESGDoc tend to be much longer and exhibit greater\nvariation in structure and presentational features.  Arguing that SOTA\napproaches are too inefficient to apply to long documents, the paper\nproposes an approach that involves applying a GNN to subgraphs that\nrepresent the local neighborhood of particular document elements\n(typically candidate headers suitable for inclusion in a TOC).  The\napproach decides what to do with each such element individually and\nconstructs the resulting TOC deterministically based on these\ndecisions.  Experiments on two problem sets, a previously proposed\ncollection of scientific documents and the corpus of the paper, show\nthat that the proposed approach outperforms a leading SOTA model, for\nwhich some of the longer documents prove to be intractable.  The\nproposed approach is shown to be superior in both efficiency and\naccuracy.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "You might want to say explicitly that you're releasing ESGDoc to\nsupport further research, rather than just \"introducing\" the dataset.\n\nYou talk about document understanding a fair amount in the front\nmatter, but never define it.  A succinct statement concerning what you\nthink research into document understanding is about (and what uses it\nmight have) would help make the scope of your research more concrete."}}]

Author rebuttal:

Meta-review: This paper presents a strong baseline dataset comparison, includes an ablation study, has a solid methodology foundation, and provides clear and concise experimental design and results. However, this paper has limitations in a lack of comparison with other state-of-the-art models, which would provide valuable insights and further strengthen the contribution of the proposed framework.
Decision: Accept-Main