SlowGuess/NAACL_2022_OCR huggingface dataset:
DYNAMICTOC: Persona-based Table of Contents for Consumption of Long Documents
Himanshu Maheshwari, Shelly Jain, Tanvi Karandikar
IIIT Hyderabad
him.mj26,shelly.jain.viii,tanvi.karandikar141@gmail.com
Nethraa Sivakumar
SSNCE, Chennai
nethraal8096@ece.ssn.edu.in
Navita Goyal
UMD, College Park
navita@umd.edu
Vinay Aggarwal
vinayagg@adobe.com
Sumit Shekhar
sushekha@adobe.com
Long documents like contracts, financial documents, etc., are often tedious to read through. Linearly consuming (via scrolling or navigation through default table of content) these documents is time-consuming and challenging. These documents are also authored to be consumed by varied entities (referred to as persona in the paper) interested in only certain parts of the document. In this work, we describe DYNAMICTOC, a dynamic table of content-based navigator, to aid in the task of non-linear, persona-based document consumption. DYNAMICTOC highlights sections of interest in the document as per the aspects relevant to different personas. DYNAMICTOC is augmented with short questions to assist the users in understanding underlying content. This uses a novel deep-reinforcement learning technique to generate questions on these persona-clustered paragraphs. Human and automatic evaluations suggest the efficacy of both end-to-end pipeline and different components of DYNAMICTOC.
Documents such as financial statements, reports and contracts are often long and comprehensive, replete with domain-specific description and information. They are meant to be consumed by several entities or personas, e.g. legal department of companies, customers or financial organizations such as banks. As these documents contain vital information about the business, the business personas are often required to read through and analyze the documents in details. These personas are often interested in different sections of the document, based on the business requirements. For example, employees might be interested in the stock programs of the company, whereas the lenders and
investors would like to read through profit statements. The traditional technology to navigate long documents is through a Table of Contents (ToC) populated with the heading of each section and chapters. However, the Table of Contents does not show the information present in the underlying paragraphs of a section, and there is no way to highlight information relevant to different personas.
To this effect, we propose DYNAMICTOC, an intelligent table of contents-based navigator. DYNAMICTOC provides user the flexibility to choose the persona and read the document from its lens. For the current work, we focus on the finance and legal domain, and hence, personas are taken as commonplace entities like investors, lenders, financial bodies, etc. DYNAMICTOC highlights the relevant sections of the document as per the persona. For this, the input finance or contract document is segmented at the paragraph level and a cluster of "aspects or topics" is inferred for each para. These are then mapped to the interest topics of the personas. Further, DYNAMICTOC has a novel question-based guided experience, to enhance the visibility of underlying information. Studies have shown that questions are more intuitive and informative than headings and hence can provide a better understanding of what the paragraph talks about. The overall interface is shown in Figure 1.
Document understanding is a critical and challenging task in information processing. There have been many related research works in this direction. Keyword detection (Liu et al., 2009; Tixier et al., 2016) & topic modeling (Blei et al., 2001) works aim is to describe the document by a few important words or topics for concise representation. The
Human Language Technologies, pages 5133 - 5143
(b) Default Table of Contents
Figure 1: A comparison of the DYNAMICTOC-based experience with the default Table of Content. DYNAMICTOC enables the user to choose a reading persona and enhances navigation through highlighting relevant sections. The relevant sections are supplemented with automatically generated questions to guide the user.
first step is to acquire a list of keyword candidates (e.g., n-grams or chunks) with heuristic methods (Hulth, 2003; Shang et al., 2018), then rank them in accordance with their importance to the document (Wu et al., 2005; Gollapalli and Caragea, 2014; Bougouin et al., 2013). Another task is compact and informative headline generation from a document (Dorr et al., 2003; Lopyrev, 2015). Text summarization is the process of generating natural language summaries from an input document retaining the most important information (Rush et al., 2015; See et al., 2017). Recently, an Outline Generation task was introduced by (Zhang et al., 2019) as a hierarchical structured prediction problem. Given a document, their aim is to first predict a sequence of section boundaries and then a sequence of section headings accordingly to come up with a Table of Contents for the same.
A related direction of work to ours is of aspect detection, which has been explored in the literature largely using user reviews for products. Early works focused on rule-based approaches using lexicons and dependency relations, and utilize manually defined rules to identify patterns and extract aspects (Qiu et al., 2011; Liu et al., 2016), which require domain-specific knowledge and human expertise. Supervised approaches formulate aspect extraction as a sequence labelling problem that can be solved by hidden Markov models (HMM) (Jin et al., 2009), conditional random fields (CRF) (Li et al., 2010; Mitchell et al., 2013; Yang and Cardie, 2012), and recurrent neural networks (RNN) (Wang et al., 2016; Liu et al., 2015). These approaches have shown better performance compared to the rule-based ones, but require large amounts of la
belled data for training. Early unsupervised systems are dominated by Latent Dirichlet Allocation (LDA)-based topic models (García-Pablos et al., 2018; Shi et al., 2018; Álvarez-López et al., 2016). Recently, deep learning based topic models (Srivastava and Sutton, 2017; Luo et al., 2019; He et al., 2017; Shi et al., 2021) have shown strong performance in extracting coherent aspects in an unsupervised manner. None of the prior works on aspect detection have worked with contracts or financial documents that are quite long (50-100 pages) in comparison to user reviews. Even if we break the document at paragraph level, it can still go over tens of lines. Hence, the importance of word frequency is much more in our case. We bridge the gap between directly using the unsupervised aspect detection frameworks for the financial documents by adding a TF-IDF based weighing parameter while training. Moreover, there are no gold standards for aspect detection for contract or finance domain, hence, we use unsupervised clustering based metrics for validating the output detection.
Further, it has been shown that question-answers play a critical role in scientific inquiry, information-seeking dialogue, and knowledge acquisition (Hintikka and Saarinen, 1979; Stede and Schlangen, 2004). In a dialogue system, question generation is used to obtain specific information from the user or make the conversation more pleasant (Shukla et al., 2019; Saeidi et al., 2018). Hence, we hypothesize that augmenting the default ToC with Questions that give a high level overview of the paragraphs can enhance the reading experience of the users. Question generation can also be seen as a
summimization or seq2seq task. Various pre-trained language models like BART (Lewis et al., 2020), PEGASUS (Zhang et al., 2020), etc., have shown excellent results for these tasks. Researchers have worked on top of these language models & proposed various rewards for QGen to optimize these models. (Kumar et al., 2019) has employed BLEU-based rewards, (Zhang and Bansal, 2019) have used answerability rewards, whereas (Xie et al., 2020) has used a combination of fluency, relevance, and answerability rewards.
Previous question generation literature has focused on generating questions based on an entity, phrase, or sentence. In this work, we explore long-form question generation, i.e., question-based long text. We explore deep reinforcement learning techniques for the same, as they have shown competitive results in various natural language generation tasks such as summarization (Pasunuru and Bansal, 2018), style transfer (Liu et al., 2021; Goyal et al., 2021), question generation (Hosking and Riedel, 2019; Xie et al., 2020) etc. Motivated by this we use BART as our base model. As there is a lack of labeled question datasets in financial domain, to overcome the domain shift problem, we train the model with additional rewards in a reinforcement learning setup to make more suitable for a general domain.
There are several commercial products for reading documents across devices, but all of them have a fixed document navigation, based on chapters and headings. To the best of our knowledge, there is no prior art looking into providing an end-to-end persona-based navigation. The mentioned technologies address only part of the required solutions. Following are the key contributions of our work:
1. We propose a novel DYNAMICTOC technology to enable persona-based non-linear navigation for efficient consumption of long documents.
2. We extend the unsupervised aspect detection to long domain-specific documents by combining TF-IDF with aspect detection process to make it more robust and show the improvements experimentally.
3. We propose a method to generate questions based on the content of the paragraph maximizing the information coverage, entity correctness & answerability of the question.
4. We showcase the viability of our pipeline and evaluate it using metric-based and a human survey-based evaluation.
SEC Filing: The SEC filing is a financial statement document submitted to the U.S. Securities and Exchange Commission. Public companies, certain insiders, and broker-dealers are required to make regular SEC filings. There are many types of documents available on the EDGAR website (Eg. 10-K, 10-Q, Form 4, etc.). For our work, we focused on the SEC 10-K documents available on the EDGAR website<sup>1</sup>. For a given company, the 10-k documents are available in HTML, XBRL and XML format. The complete submission text file for a 10-k document (XML) was used for parsing. We split the document content into different items ranging from item 1 to item 16. These 10-K documents range from 50 to 120 pages and contain multiple tables along with text paragraphs.
ELI5: We use the ELI5 dataset (Fan et al., 2019) to train the question generation model. ELI5 or Explain Like I'm 5, is a question-answer dataset scraped from the subreddit r/explainlikeimfive/. The subreddit rules encourage people to ask a question about any topic and get an answer for it. To maintain the dataset's quality, we only select those question-answer pairs with more than two upvotes. Note that no dataset for such question generation task exists for contractual and financial documents. Hence, we resort to use the ELI5 dataset for supervised training and use that model for inferencing on documents from a different domain.
DATASET	TRAIN SIZE	TEST SIZE
ELI5	100,000	10,000
r/AskLegal	-	10,067
r/AskEconomics	-	98
Table 1: Train/Test Statistics of question-answer pairs for the datasets used.
To test the model's performance on the domain-specific dataset, we also scrape question-answer pairs from two different subreddits - r/AskLegal and r/AskEconomics. As the name suggests, they contain questions (and answers) from the legal and economics domain respectively. Table 1 includes the statistics of the three datasets.
<sup>1</sup>https://www.sec.gov/edgar.shtml
In this section, different components of the DYNAMICTOC are described in details.
4.1 Aspect Detection
Aspect detection has been popularly used with analysing user reviews to understand their preferences. We leverage an unsupervised technique for aspect detection and extend it to a new use case - for the modelling of user profiles from a given document and using this info for segregating the document text based on the determined aspects. Data Pre-processing: For the input SEC filings, text corresponding to each paragraph is obtained and considered as a separate data point. Extra information such as headings, sub-headings, blank lines, signature fields etc. are discarded. Along with this, any paragraph with less than 10 words are discarded. The text is pre-processed before training the model for aspect detection. We require three formatting styles for each paragraph which is consolidated in a single dictionary. (1) Tokenized words converted to lowercase characters. (2) The word stems of the text which has been lower-cased and tokenised. (3) The content words (meaningful words, like nouns, verbs, adjectives and adverbs) of the paragraph.
Proposed Asp-SSCL Method: Aspect detection aims at extracting interpretable aspects from the textual documents without human supervision. We propose an approach, Asp-SSCL based on self-supervised contrastive learning framework by (Shi et al., 2021) for aspect detection. We use the following steps for aspect detection from the contractual and financial documents:
1. Vocabulary formation and IDF indexing: First, we obtain a vocabulary for the whole corpus. This is sorted alphabetically and each word is given an index, so that corresponding IDF/word vectors can be easily referenced. 128-dimensional word vectors are generated on the corpus by a skip-gram model with an n-gram size of 5.
2. Weak Mapping: Prior aspect detection methods require a gold set labels for validating aspect model training, either through human supervision or rules-based mapping to gold set keywords. However, as no such gold aspect labels exist for our case, we first use text embedding via sentence transformers<sup>2</sup>. These are then clustered using K-means to
obtain 20 clusters comprising of 10 keywords each, which are used for aspect mapping.
3. Contrastive Learning: The mapping and generated word vectors are then used for training the self-supervised contrastive learning method, (Shi et al., 2021), which outputs the final aspect clusters and keywords.
4. TF-IDF Weighing: Further, since these documents are text-heavy, we introduce a modification, Asp-SSCL-TFIDF which includes TF-IDF weighing term in the original implementation, to ensure rare but relevant words are considered as important as opposed to more frequently occurring words. Each word representation is modified by multiplying it with the TF-IDF score so that the algorithm can adapt to the financial corpus better.
4.2 Persona Mapping
The aspects generated on the corpus are used as dimensions that define the document. Each persona is expected to be interested in one or more of these dimensions. We call the mapping between multiple personas and multiple aspects as the persona space.
We consulted a domain expert (financial domain; specifically for SEC 10-K filings) to create a matrix of personas, who read such documents, and what kind of information they are interested in. Figure 2 lists out the various stakeholders of a general 10-K filing against the different sections of the document each stakeholder is interested in. The stakeholders are grouped together to form the personas used in DYNAMICTOC, viz. employees, business partners, investors and lenders, financial bodies and advisory and regulatory firms. Similarly, the columns (headings) are grouped together according to similarity to create a mapping of topics of interest for each persona.
We can map these columns to the aspects we get from the Aspect Detection Module and determine if a particular persona is interested in that paragraph or not. For this, the aspects obtained from the unsupervised technique are compared against the simplified column values from the constructed matrix. The columns with the greatest similarity (above a threshold) are associated with each persona. For getting the personas interested in each paragraph, the paragraphs are first tagged for aspect. From the resultant vector (which represents the confidence score of the text for each aspect), the combined score for each persona is calculated using the scores of its constituent aspects. This
$^{2}$ https://www.sbert.net/
Figure 2: Table showing different stakeholders who would be interested in consuming SEC filings and corresponding sections of their interests. The rows are grouped together to form the five personas used in the work. Similar column topics have also been grouped for aspect mapping.
is used to segregate the paragraphs for enhanced document consumption.
Note that for financial documents, we were able to gather domain knowledge and leverage it to obtain the persona space. But the proposed technique is generalizable to other domains as well. In the absence of domain-specific knowledge, each aspect is a sufficiently distinct topic and can be treated as a proxy to personas. Hence, modelling of interests can be done directly on the basis of aspects in such cases.
4.3 Intelligent Navigation via Question Generation
It has been shown that question-answers play a critical role in scientific inquiry, information-seeking dialogue, and knowledge acquisition (Hintikka and Saarinen, 1979; Stede and Schlangen, 2004). Additionally, unstructured lists of "Frequently Asked Questions (FAQs)" are regularly deployed at scale to present information. On top, questions can provide a meaningful understanding of the document at a paragraph or section level which cannot be directly captured by a heading (or subheading). Therefore, to aid in document consumption, we generate long-form questions (i.e., questions based on paragraphs instead of entities) to enhance the navigation experience.
Model Architecture: We use an encoder-decoder architecture for the task of generating questions given the paragraph as context which essentially is a sequence-to-sequence task. Large pretrained language models like BART (Lewis et al., 2020), PEGASUS (Zhang et al., 2020), etc., have shown excellent results in summarization tasks. Motivated by this, we employ BART as our under
lying language model. The task is to generate questions covering the entire paragraph and summarize it capturing the most-salient information in form of a question. The BART model has shown promising results in abstractive summarization tasks, making it a natural choice.
We use ELI5 (Fan et al., 2019) dataset for training the model. The answer is provided as the input to the encoder-decoder model which is trained to generate the corresponding question and minimize the cross-entropy loss with respect to the ground truth. Although such supervised training is straight-forward, due to the domain shift from ELI5 to financial language, qualitative evaluations showed that the model produced some irrelevant questions, some entities were artificially induced (that it might have seen during the training time) and sometimes, it could not cover the entire paragraph. Hence, we augmented the vanilla BART with three additional rewards targeting the qualities we seek in the final generated questions. We call the resulting model Variant BART. Figure 3 shows the proposed pipeline. The following sections explain these rewards in detail:
Figure 3: Training Variant BART for Question Generation with feedback rewards
- Answerability Classifier Reward: Qualitative evaluations showed some of the questions generated were not answerable by the paragraph, making them unsuitable for the task of understanding the paragraph easily. To address this, we trained a classifier to judge the answerability of the question given the paragraph. Basically, the paragraph and the generated question would be fed as input to the classifier and the classifier predicts "1" if the question is answerable by the para, otherwise "0". The classifier is a fine-tuned Roberta model. We create data with both positive and negative samples to fine tune the Roberta model. We use the following strategy to create the training data for the binary classifier: (a) We select 10,000 random question-answer pairs from the ELI5 dataset. Note that in ELI5 data, some questions have multiple answer paras too. All these forms our positive samples. To create negative samples for a question  $Q'$ , we take its corresponding answer para,  $A'$ , and a random set of 100 question-answer pairs. We compare the similarity of the answer para ( $A'$ ) with all the 100 answers. Top 3 most similar answers are taken as the negative samples for the question  $Q'$ . (b) Given that style of ELI5 answers and Wikipedia paragraphs are very similar $^3$ , we create negative samples in another way as well to introduce diversity. Wikipedia articles are chosen based on their similarity with ELI5 data and for each question, we compute the similarity score with each para in the wiki articles. The topmost similar paras are of interest, and we sample 10 most similar paras out of top 20 and call them our negative samples. Table 2 mentions the data statistics. The reward is computed using the equation:
$R_{answerability} = Prob_{classifier}(1|P,Q)$
DATASET	TRAIN	TEST
#Unique Questions	10,000	1,000
Total Samples	208,871	21,045
Positive Samples	30,564	2,974
Negative Samples	178,307	18,071
Neg to Pos Ratio	5.83	6.04
Table 2: Dataset stats to train answerability classifier
- Entity Correctness Reward: The vanilla BART model can generate questions with
hallucinated entities and names like Microsoft, Apple etc. even when there was no mention of them in the corresponding paragraph. To tackle this, we identify the named entities present in the generated question. If those entities appear in the passage, we give a reward of 1 else 0. If there is no entity in the generated question, a reward of 0.5 is given to the model. Mathematically, reward is given by:
$$
R _ {e n t i t y} = \left\{ \begin{array}{l l} 1, & \text {i f} e (Q) \neq \phi , e (Q) \subseteq e (P) \\ 0, & \text {i f} e (Q) \neq \phi , e (Q) \not \subseteq e (P) \\ 0. 5, & \text {i f} e (Q) = \phi \end{array} \right.
$$
where,  $e(\cdot)$  denotes the entities in the question or paragraph.
- Coverage Reward: We observe that the output question did not cover the entire information present in the paragraph and instead focused on certain segments of it. We introduce this reward to improve information coverage. The idea is similar to the entity correctness reward. We first identify keywords from the paragraph using YAKE algorithm (Campos et al., 2018). Then we calculate the similarity of the generated question with these keywords. We use the Extended String Subsequence Kernel (ESSK) introduced in (Hirao et al., 2003) to calculate this similarity score. The idea is that YAKE would generate keywords from different parts of the paragraph. When we calculate the similarity of this keyword list with the generated question, we are encouraging the model to cover the entire paragraph. Thus, given a passage P and the generated question Q, the reward R is defined as follows:
$$
R _ {c o v e r a g e} = E S K (Y A K E (P), Q)
$$
(Lai et al., 2021) shows how to use rewards on top of language models for policy learning. We adopt the same setup. The policy gradient,  $\nabla_{\phi}J(\phi)$  is given by:
$$
\nabla_ {\phi} J (\phi) = E \left[ R \cdot \nabla_ {\phi} \log \left(P \left(y ^ {s} \mid x, \phi\right)\right) \right] \tag {1}
$$
where,  $R$  denotes reward value,  $\phi$  represents the model parameters,  $x$  is the input paragraph and  $y^{s}$  is obtained by greedily maximizing the distribution of BART outputs at each timestep. Hence, the
<sup>3</sup>https://yjernite.github.io/lfqa.html
overall loss term for training the proposed Variant BART model becomes:
$$
L _ {\text {t o t a l}} = \lambda_ {C E} \cdot L _ {C E} + \lambda_ {\text {r e w a r d}} \cdot L _ {\text {r e w a r d}} \tag {2}
$$