SlowGuess/NAACL_2022_OCR huggingface dataset:
In order to evaluate the different parts of the pipeline, we employ metric based evaluation schemes. Since there is no off-the-shelf criterion to evaluate all the stages of the pipeline together, we have provided independent evaluations for each of the sub-modules. However, the intended goal of our pipeline is to facilitate the readers in consuming long documents through the lens they deem most suitable for them. To facilitate an end-to-end evaluation and to understand whether the persona-based document segmentation with enhanced Table of Content is informative or not, we have conducted a small scale human evaluation. Both metric-based and human-based evaluations are discussed in the following subsections.
5.1 Metric-Based Evaluation
Aspect Detection: Figure 4 shows the t-SNE clustering of the outputs of Asp-SSCL, Asp-SSCL-TFIDF, that are determined from the SEC-10K filing corpus. For a baseline comparison, we also plot the output for LDA (10 clusters) for the corpus. Essentially, each cluster is a bag of words indicating some vital theme that is mentioned in the corpus. We would want the clusters to be as independent from each other as possible as that would mean different kinds of information is captured by different clusters with minimal overlap. We observe that adding TF-IDF scores to the aspect detection module helps as the clusters' separation gets better as shown in the Figure 4. Further, for the baseline using LDA, the separation is not clear. Some of the examples of cluster keywords are shown in Figure 5.
Question Generation: Since no ground truth questions are available for the SEC-10K Filing dataset, we report the following evaluations for the question generation module. First, we report the "type" of questions that are generated using the Vanilla BART model and the Variant BART model that is trained with a combination of the rewards we added on top of it (Table 3).
On analysing this table, we see that "What" questions are heavily generated on the 10K filing data.
(a) Asp-SSCL
(b) Asp-SSCL with TF-IDF
(c) LDA
Figure 4: Aspect clusters using (a) default Asp-SSCL and (b) Asp-SSCL with TF-IDF and (c) LDA
LDA cluster words:
1. cost, fiscal year, revenue increase, expense, sale, million, primarily due
2. financial statement, report, control, intern, consolidate, account, audit, inform
3. rate, interest, value, invest, market, currency, hedge, risk, fair instrument
4. stock, share, common award, grant, option, incentive, per companies, date
Asp-SSCL cluster words:
1. vest, vesting, shares, vested, awards, bonus, equal, grant, month
2. regulations, laws, federal, act, rules, regulation, legislation, requirements, privacy, law
3. opportunities, strategy, initiatives, strategies, efficiencies, efforts focus opportunity, objectives
4. deferred, tax, unrecognized differences, recorded, depreciation, taxes, income, accumulated expense
Asp-SSCL-TFIDF cluster words:
1. damage, breaches, injury, disasters, breach, fines, damages, disruptions, failures, sanctions
2. assumptions, forward looking, estimates, inputs, hierarchy, qualitative, judgment, analysis
3. shares, awards, common, options, vesting, stock, voting, vested, award, vest
4. allowance, tax allowances, losses, adjustments, charges, deferred, differences, taxable, carry forwards
Figure 5: Examples of cluster words generated using LDA, Asp-SSCL and Asp-SSCL with TF-IDF.
Q TYPE	ELI5 TRAIN SET (%)	VANILLA BART	VARIANT BART
What	12.93	54.28	44.75
Where	0.55	0.0	0.56
Why	36.23	11.47	11.93
How	21.88	30.18	37.44
Who	0.28	3.2	3.9
When	1.62	0.33	0.84
Other	26.51	0.54	0.56
Table 3: Type of questions generated by Vanilla BART & Variant BART on an SEC filing.
This suggests that the nature of 10K filing is such that the question asked about them is "What" type. We also observe that biases of training data are not creeping in the model, as the percentage of the "What" questions in the training dataset is four times less than the model's output. Similarly, the percentage of other questions is significant in the ELI5 dataset but is very small in our model's output. Thus, we can safely say that the model learns to generate questions and not mimic the ELI5 dataset.
Although we don't have the gold corpus for SEC filing dataset, we evaluate the performance of question generation model on the AskLegal and AskEconomics subreddits since we have the ground
4https://lvdmaaten.github.io/tsne/
truth questions for them. We report the BLEU and ROUGE scores that are standard metrics in Natural Language Processing literature and are a measure of overlap or common n-grams between the generated text and the ground truth. We also report the answerability score by feeding the generated question and input para to the classifier we trained (as mentioned in Reward 1 – Question Generation Section). Table 4 shows the corresponding results.
MODEL	BLEU	ROUGE-L	ANSWERABILITY
Vanilla (r/AskLegal)	0.274	0.278	95.92
Variant (r/AskLegal)	0.264	0.240	98.98
Vanilla (r/AskEconomics)	0.289	0.298	95.16
Variant (r/AskEconomics)	0.291	0.306	96.17
Table 4: Performance comparison of Vanilla and Variant BART on r/AskLegal and r/AskEconomics subreddits
QUESTION	AVG SCORE (1-5)
How often do you come across a long document?	2.83
How often do you use a document reader?	4.33
How satisfied are you with the 'Default' reading experience?	3.00
How would you rate the option to choose persona as an aid in reading the document?	4.50
How would you rate jumping to relevant parts of the document?	4.16
How would you rate the utility of presented questions?	4.40
Table 5: Results from the human experiment on using the Default Reading experience with DYNAMICToC.
On closely analysing the paragraph, reference question, and the generated question, we observe the following three things: (i) There is more than one way to ask the same question and there could be multiple questions around the same topic. (ii) Input paragraphs may have more than one prominent topics. The generated question might be focused on one such topic, and the reference question is focused on another. (iii) Some answers/passages are unrelated to the question or require some background, and thus, the generated questions are very different from the actual question.
The above reasons explain the fluctuation in scores for Vanilla and Variant BART, and thus the answerability of the generated question becomes an important metric. The variant model trained with additional rewards has the highest answerability score across all the datasets. This suggests that including a coverage-based loss not only helps cover the information of the entire paragraph but also helps increase the generated question's answerability as different themes of the passages are covered.
We conducted a small human evaluation involving 8 participants (age -  $27.8 \pm 6.7$ , 2 females). The participants were technology workers internal to our organization. They were asked to play around with a web demo to experience the DYNAMICTOC, for different SEC filings. They were first shown the default section heading-based reading experience and then they choose the type of persona as whom they wish to consume the document. After this, they filled a questionnaire about their experience. The results of the survey are summarized in Table 5.
Some relevant comments from the survey are as follows - (i) How do we ensure that all the relevant information will be covered by the sections highlighted as important for a particular "persona"? (ii) Although questions generated are relevant, some of the why questions are not answered by the paragraphs they point to. (iii) Interesting experiment with possibly multiple use-cases. The first comment is actually true for all summarization tasks, hence, DYNAMICTOC does not disrupt the linear flow. The second feedback indicates scope for further research in the question generation space. The overall response is immensely positive and the scores of 4.50, 4.16 and 4.40 in Table 5 reflect the same.
In this work, we have proposed a novel DYNAMICTOC framework for consumption of long documents. Financial documents are high value documents for businesses, and are often long and complex. The default ToC-based reading experience is quite limited and document consumption can be enhanced using intelligent technologies. DYNAMICTOC is one of the first works to pursue this exciting research direction. DYNAMICTOC would benefit from in-domain learning of aspect keywords and questions. Evaluation of paragraph segmentation and mapping of personas to the aspects are future directions. A better understanding of personas would generalize the work to different domains.
Tamara Álvarez-López, Jonathan Juncal-Martínez, Milagros Fernández-Gavilanes, Enrique Costa-Montenegro, and Francisco Javier González-Castano. 2016. GTI at SemEval-2016 task 5: SVM and
CRF for aspect detection and unsupervised aspect-based sentiment analysis. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 306-311, San Diego, California. Association for Computational Linguistics.
David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2001. Latent dirichlet allocation. In Advances in Neural Information Processing Systems 14 [Neural Information Processing Systems: Natural and Synthetic, NIPS 2001, December 3-8, 2001, Vancouver, British Columbia, Canada], pages 601-608. MIT Press.
Adrien Bougouin, Florian Boudin, and Beatrice Daille. 2013. TopicRank: Graph-based topic ranking for keyphrase extraction. In Proceedings of the Sixth International Joint Conference on Natural Language Processing, pages 543-551, Nagoya, Japan. Asian Federation of Natural Language Processing.
Ricardo Campos, Vitor Mangaravite, Arian Pasquali, Alipio Mario Jorge, Celia Nunes, and Adam Jatowt. 2018. Yake! collection-independent automatic keyword extractor. In European Conference on Information Retrieval, pages 806-810. Springer.
Bonnie Dorr, David Zajic, and Richard Schwartz. 2003. Hedge trimmer: A parse-and-trim approach to headline generation. In Proceedings of the HLT-NAACL 03 Text Summarization Workshop, pages 1-8.
Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. 2019. ELI5: Long form question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3558-3567, Florence, Italy. Association for Computational Linguistics.
Aitor García-Pablos, Montse Cuadros, and German Rigau. 2018. W2vlda: almost unsupervised system for aspect based sentiment analysis. Expert Systems with Applications, 91:127-137.
Sujatha Das Gollapalli and Cornelia Caragea. 2014. Extracting keyphrases from research papers using citation networks. In Proceedings of the Twenty-Eighth AAAI Conference on Artificial Intelligence, July 27-31, 2014, Quebec City, Quebec, Canada, pages 1629-1635. AAAI Press.
Navita Goyal, Balaji Vasan Srinivasan, Anandhavelu N, and Abhilasha Sancheti. 2021. Multi-style transfer with discriminative feedback on disjoint corpus. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3500-3510, Online. Association for Computational Linguistics.
Ruidan He, Wee Sun Lee, Hwee Tou Ng, and Daniel Dahlmeier. 2017. An unsupervised neural attention model for aspect extraction. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages
388-397, Vancouver, Canada. Association for Computational Linguistics.
Jaakko Hintikka and Esa Saarinen. 1979. Information-seeking dialogues: Some of their logical properties. Studia Logica, 38(4):355-363.
Tsutomu Hirao, Jun Suzuki, Hideki Isozaki, and Eisaku Maeda. 2003. Ntt's multiple document summarization system for duc2003. In Proc. DUC.
Tom Hosking and Sebastian Riedel. 2019. Evaluating rewards for question generation models. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2278-2283, Minneapolis, Minnesota. Association for Computational Linguistics.
Anette Hulth. 2003. Improved automatic keyword extraction given more linguistic knowledge. In Proceedings of the 2003 Conference on Empirical Methods in Natural Language Processing, pages 216-223.
Wei Jin, Hung Hay Ho, and Rohini K Srihari. 2009. Opinionminer: a novel machine learning system for web opinion mining and extraction. In Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 1195-1204.
Vishwajeet Kumar, Ganesh Ramakrishnan, and Yuan-Fang Li. 2019. Putting the horse before the cart: A generator-evaluator framework for question generation from text. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 812-821, Hong Kong, China. Association for Computational Linguistics.
Huiyuan Lai, Antonio Toral, and Malvina Nissim. 2021. Thank you BART! rewarding pre-trained models improves formality style transfer. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 484-494, Online. Association for Computational Linguistics.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computational Linguistics.
Fangtao Li, Chao Han, Minlie Huang, Xiaoyan Zhu, Ying-Ju Xia, Shu Zhang, and Hao Yu. 2010. Structure-aware review mining and summarization. In Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010), pages 653-661, Beijing, China. Coling 2010 Organizing Committee.
Feifan Liu, Deana Pennell, Fei Liu, and Yang Liu. 2009. Unsupervised approaches for automatic keyword extraction using meeting transcripts. In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 620-628, Boulder, Colorado. Association for Computational Linguistics.
Pengfei Liu, Shafiq Joty, and Helen Meng. 2015. Fine-grained opinion mining with recurrent neural networks and word embeddings. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1433-1443, Lisbon, Portugal. Association for Computational Linguistics.
Qian Liu, Bing Liu, Yuanlin Zhang, Doo Soon Kim, and Zhiqiang Gao. 2016. Improving opinion aspect extraction using semantic similarity and aspect associations. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, February 12-17, 2016, Phoenix, Arizona, USA, pages 2986-2992. AAAI Press.
Yixin Liu, Graham Neubig, and John Wieting. 2021. On learning text style transfer with direct rewards. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4262-4273, Online. Association for Computational Linguistics.
Konstantin Lopyrev. 2015. Generating news headlines with recurrent neural networks. arXiv preprint arXiv:1512.01712.
Ling Luo, Xiang Ao, Yan Song, Jinyao Li, Xiaopeng Yang, Qing He, and Dong Yu. 2019. Unsupervised neural aspect extraction with sememes. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019, pages 5123-5129. ijcai.org.
Margaret Mitchell, Jacqui Aguilar, Theresa Wilson, and Benjamin Van Durme. 2013. Open domain targeted sentiment. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1643-1654, Seattle, Washington, USA. Association for Computational Linguistics.
Ramakanth Pasunuru and Mohit Bansal. 2018. Multireward reinforced summarization with saliency and entailment. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 646-653, New Orleans, Louisiana. Association for Computational Linguistics.
Guang Qiu, Bing Liu, Jiajun Bu, and Chun Chen. 2011. Opinion word expansion and target extraction through double propagation. Computational Linguistics, 37(1):9-27.
Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 379-389, Lisbon, Portugal. Association for Computational Linguistics.
Marzieh Saeidi, Max Bartolo, Patrick Lewis, Sameer Singh, Tim Rocttäschel, Mike Sheldon, Guillaume Bouchard, and Sebastian Riedel. 2018. Interpretation of natural language rules in conversational machine reading. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2087-2097, Brussels, Belgium. Association for Computational Linguistics.
Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073-1083, Vancouver, Canada. Association for Computational Linguistics.
Jingbo Shang, Jialu Liu, Meng Jiang, Xiang Ren, Clare R Voss, and Jiawei Han. 2018. Automated phrase mining from massive text corpora. IEEE Transactions on Knowledge and Data Engineering, 30(10):1825-1837.
Tian Shi, Kyeongpil Kang, Jaegul Choo, and Chandan K. Reddy. 2018. Short-text topic modeling via non-negative matrix factorization enriched with local word-context correlations. In Proceedings of the 2018 World Wide Web Conference on World Wide Web, WWW 2018, Lyon, France, April 23-27, 2018, pages 1105-1114. ACM.
Tian Shi, Liuqing Li, Ping Wang, and Chandan K Reddy. 2021. A simple and effective self-supervised contrastive learning framework for aspect detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 13815-13824.
Pushkar Shukla, Carlos Elmadjian, Richika Sharan, Vivek Kulkarni, Matthew Turk, and William Yang Wang. 2019. What should I ask? using conversationally informative rewards for goal-oriented visual dialog. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6442-6451, Florence, Italy. Association for Computational Linguistics.
Akash Srivastava and Charles Sutton. 2017. Autoencoding variational inference for topic models. In 5th International Conference on Learning Representations, ICLR 2017, Toulouse, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net.
Manfred Stede and David Schlangen. 2004. Information-seeking chat: Dialogues driven by topic-structure. In Proceedings of Catalog (the 8th workshop on the semantics and pragmatics of dialogue; SemDial04). CiteSeer.
Antoine Tixier, Fragkiskos Malliaros, and Michalis Vazirgiannis. 2016. A graph degeneracy-based approach to keyword extraction. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1860-1870, Austin, Texas. Association for Computational Linguistics.
Wenya Wang, Sinno Jialin Pan, Daniel Dahlmeier, and Xiaokui Xiao. 2016. Recursive neural conditional random fields for aspect-based sentiment analysis. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 616-626, Austin, Texas. Association for Computational Linguistics.
Yi-fang Brook Wu, Quanzhi Li, Razvan Stefan Bot, and Xin Chen. 2005. Domain-specific keyphrase extraction. In Proceedings of the 14th ACM international conference on Information and knowledge management, pages 283-284.
Yuxi Xie, Liangming Pan, Dongzhe Wang, Min-Yen Kan, and Yansong Feng. 2020. Exploring question-specific rewards for generating deep questions. In Proceedings of the 28th International Conference on Computational Linguistics, pages 2534-2546, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Bishan Yang and Claire Cardie. 2012. Extracting opinion expressions with semi-Markov conditional random fields. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 1335-1345, Jeju Island, Korea. Association for Computational Linguistics.
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu. 2020. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. In International Conference on Machine Learning, pages 11328-11339. PMLR.
Ruqing Zhang, Jiafeng Guo, Yixing Fan, Yanyan Lan, and Xueqi Cheng. 2019. Outline generation: Understanding the inherent content structure of documents. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2019, Paris, France, July 21-25, 2019, pages 745-754. ACM.
Shiyue Zhang and Mohit Bansal. 2019. Addressing semantic drift in question generation for semi-supervised question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2495-2509, Hong Kong, China. Association for Computational Linguistics.