SlowGuess/NAACL_2021_OCR huggingface dataset:
In this section, we describe the performance of our Nutribullet Hybrid system and baselines on summarization and summary updates. We report empirical results, human evaluation and present sample outputs, highlighting the benefits of our method.
Single and Multi-issues Summarization: We describe the results on the task of generating summaries. Table 3 presents the automatic evaluation results for the food and single issue summarization task. High KG(I) and KG(G) scores for our method indicate that the generated text is faithful to input entities and relevant. In particular, a high Aggregation Cognisance (Ag) score indicates that our model generates summaries which are cog
7We run entity tagging plus relation classification on top of the model output and gold summaries. We match the gold  $(e_i^g,e_j^g,r^g)$  tuples using word embedding based cosine similarity with the corresponding entities in the output structures  $(e_i^o,e_j^o,r^o)$ . A cosine score exceeds a threshold of 0.7 is set (minimize false positives) to identify a match.
Transformer (baseline)
* Whole - grain cereals may protect against obesity, diabetes and certain cancers. However, more research is needed.
* Whole grains, such as mozambican grass, are safe to eat with no serious side effects.
* Whole - grain cereals may protect against obesity, diabetes and certain cancers. However, more research is needed.
* Whole grains, such as blueberries, are likely safe to eat with no serious side effects.
* Whole grains are safe to eat. However, people with type 2 diabetes should avoid whole grains.
* Whole grains are lower in carbs than whole grains, making them a good choice for people with type 2 diabetes.
Our Method
* Whole grains has been shown to lower weight gain and improve various type 2 diabetes risk factors.
* Whole grains has been shown to lower insulin resistance and improve various cancer risk factors.
* Whole grains has been linked to several other potential health benefits, such as improved CVD risk, eyesight, and memory. However, more studies are needed to draw stronger conclusions.
* There is some evidence, in both animals and humans, that whole grains can reduce mortality by regulating the hormone ghrelin.
Table 4: Example outputs of our model and the Transformer baseline for a multi-issues summary. Trained on limited parallel data, the Transformer baseline produces repetitive text with factual inaccuracies, while our method is able to provide more accurate and diverse summarization.
Model	KG(G)	KG(I)
Copy-gen	0.43	0.69
Transformer	0.33	0.73
Ours	0.5	0.90
nizant of the varying degrees of consensus in the input Pubmed documents. Compared to other baselines we also receive a competitive score on the automatic Rouge metric, beating Copy-gen, Entity Data2text and GraphWriter baselines while falling short (by  $1.7\%$ ) of the Transformer baseline. The baselines, especially Transformer, tend to produce similar outputs for different inputs (see Table 4). Since a lot of these patterns are learned from the human summaries, Transformer receives a high Rouge score. However, as in the low resource regime, the baseline does not completely capture the content and aggregation, it fails to get a very high KG(G) or Ag score. A similar trend is observed for the other baselines too, which in this low resource regime produce a lot of false information, reflected in their low KG(I) scores.
Human evaluation, conducted by considering scores, on a 1-4 Likert scale, from three annotators for each instance, shows the same pattern. Our model is able to capture the most relevant information, when compared against the gold summaries while producing fluent summaries. The Transformer baseline produces fluent summaries, which are not as relevant. The performance is poorer for the Copy-gen, Entity Data2text and GraphWriter models.
In the multi-issues setting, the baselines access
the gold annotations with respect to the input documents' clustering. Our model conducts the extra task of grouping the selected tuples, using the "New List" action. Our model performs better than the baselines on both the KG(I) and KG(G) metrics as seen in Table 5. Again, the pattern of producing very similar and repetitive sentences hurts the baselines. They fail to cover different issues and tend to produce false information, in this low resource setting. Our model scores an  $7\%$  higher on KG(G) and  $17\%$  higher on KG(I) compared to the next best performance, in absolute terms. Table 4 shows the comparison between the outputs produced by our method and the Transformer baseline on the benefits of whole-grains. Our method conveys more relevant, factual and organized information in a concise manner.
Table 5: KG in gold(G) and KG in input(I) in our model and baselines in the food and multi-issues setting. The best results are in bold.
Model	Fusion Update KG(G)	Contradictory Update Ag
Copy-gen	0.16	0.50
GraphWriter	0.0	0.50
Entity Data2text	0.16	0.50
Transformer	0.16	0.46
Ours	0.33	0.76
Table 6: The middle column shows KG in gold(G) in our model and baselines for fusion updates. The last column shows Aggregation Cognisance (Ag) in our model and baselines in the contradictory update setting. The best results are in bold.
Summary Update: We study the efficacy of our model to fuse information in existing summaries on receiving new Pubmed studies. As the KG(G) metric in 6 shows, our model is able to select and fuse more relevant information. Table 7 shows two examples of summaries on flaxseeds where our model successfully fuses new information.
Table 6's last column presents the automatic
Old Summary	Flax seeds contain a group of nutrients called lignans , which have powerful antioxidant and estrogen properties .
New Inputs	(i):"...current overall evidence indicates that FS and its components are effective in the risk reduction and treatment of breast cancer and safe for consumption by breast cancer patients..." (ii): "...Consumption of flaxseed was associated with a significant reduction in breast cancer risk as was consumption of flax bread ..." (iii): "...a flaxseed-supplemented, fat-restricted diet may affect the biology of the prostate and associated biomarkers..."
Copy-gen	Avocados may help fight cancer risk, boost inflammation. In a pasteurized called polyphenols, which may aid weight loss.
Transformer	Flaxseed oil is high in antioxidants that may help reduce the risk of several chronic diseases .
Ours	Flax seeds are rich in antioxidant , especially through lignans. They contain beneficial nutrients which can help protect your body against certain types of breast cancer .
Old Summary	Flax seeds, high in fiber, can be a beneficial addition to the diet of people with diabetes .
New Input	"...showed fasting blood sugar in the experimental group decreased...the total cholesterol reduced...Results showed a decrease in low-density lipoprotein cholesterol...The study demonstrated the efficacy of flax gum in the blood biochemistry profiles of type 2 diabetes."
Copy-gen	Eating apart has been linked to increased growth cholesterol, and cholesterol levels. However, more studies are needed to confirm possible effect.
Transformer	Flaxseed extract may help lower blood sugar levels .
Ours	Flax seeds are high in fiber , which is beneficial for people with diabetes and associated with a reduced low-density lipoprotein cholesterol .
evaluation results to demonstrate the efficacy of maintaining Aggregation Cognisance (Ag), which is critical when updating summaries on receiving contradictory results. The high performance in this update setting demonstrates the Surface Realization model's ability to produce aggregation cognizant outputs, in contrast to the baselines that do not learn this reasoning in a low resource regime.
Analysis: Information Extraction and Content Aggregation Information extraction is the critical first step performed for the input documents in order to get symbolic data for content selection and aggregation. To this end, we report the performance of the information extraction system, which is composed of two models - entity extraction and relation classification. As reported in Table 8, the entity extraction model, a crf-based sequence tagging model, receives a token-level F1 score of  $79\%$ . The relation classification model, a BERT based text classifier, receives an accuracy of  $69\%$ .
The performance of the information extraction models is particularly important for the content aggregation sub-task. In order to analyse this quantitatively, we perform manual analysis of the 179 instances in the dev set and compare them to the system identified aggregation – information extraction followed by the deterministic rules in Table 1. Given the simplicity of our rules, system's  $78\%$  accuracy in Table 8 is acceptable. Deeper analysis shows that the performance is lowest for Population Scoping and Contradiction with an accuracy of  $52\%$  and  $56\%$  respectively. The performance
of Population Scoping being low is down predominantly to the simplicity of the rules. Most mistakes occur when the input studies are review studies that don't mention any population but analyze results from several past work. Contradiction suffers because of the information extraction system and stronger models for the same should be able to alleviate the errors.
Table 7: Example outputs of our model and baselines for a summary update upon receiving new information about flaxseeds + cancer and flaxseeds + cholesterol, respectively. Our model maintains old information and updates accurately. In the cholesterol case, Transformer adds new information but misses the old information.
Task	Performance
Entity Extraction	0.79
Relation Classification	0.69
Aggregation Operator Identification	0.78
Table 8: Performance of our information extraction system and its impact on content aggregation.
While modern models produce fluent text in multi-document summarization, they struggle to capture the consensus amongst the input documents. This inadequacy – magnified in low resource domains, is addressed by our model. Our model is able to generate robust summaries which are faithful to content and cognizant of the varying consensus in the input documents. Our approach is applicable in summarization and textual updates. Extensive experiments, automatic and human evaluation underline its impact over state-of-the-art baselines.
We thank the MIT NLP group and the reviewers for their helpful discussion and comments. This work is supported by DSO grant DSOCO1905.
Gerald Albaum. 1997. The likert scale revisited. _Market Research Society_. Journal., 39(2):1-21.
Reinald Kim Amplayo and Mirella Lapata. 2019. Informative and controllable opinion summarization. arXiv preprint arXiv:1909.02322.
Regina Barzilay, Daryl McCullough, Owen Rambow, Jonathan DeCristofaro, Tanya Korelsky, and Benoit Lavoie. 1998. A new approach to expert system explanations. In Natural Language Generation.
Tal Baumel, Matan Eyal, and Michael Elhadad. 2018. Query focused abstractive summarization: Incorporating query relevance, multi-document coverage, and summary length constraints into seq2seq models. arXiv preprint arXiv:1801.07704.
Jianpeng Cheng and Mirella Lapata. 2016. Neural summarization by extracting sentences and words. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 484-494, Berlin, Germany. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
Chris Donahue, Mina Lee, and Percy Liang. 2020. Enabling language models to fill in the blanks. arXiv preprint arXiv:2005.05339.
Alexander Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir Radev. 2019. Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1074-1084, Florence, Italy. Association for Computational Linguistics.
Angela Fan, David Grangier, and Michael Auli. 2017. Controllable abstractive summarization. arXiv preprint arXiv:1711.05217.
Kelvin Guu, Tatsunori B. Hashimoto, Yonatan Oren, and Percy Liang. 2018. Generating sentences by editing prototypes. Transactions of the Association for Computational Linguistics, 6:437-450.
Kelvin Guu, Panupong Pasupat, Evan Liu, and Percy Liang. 2017. From language to programs: Bridging reinforcement learning and maximum marginal likelihood. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics
(Volume 1: Long Papers), pages 1051-1062, Vancouver, Canada. Association for Computational Linguistics.
Andrew Hoang, Antoine Bosselut, Asli Celikyilmaz, and Yejin Choi. 2019. Efficient adaptation of pretrained transformers for abstractive summarization. arXiv preprint arXiv:1906.00138.
Rik Koncel-Kedziorski, Dhanush Bekal, Yi Luan, Mirella Lapata, and Hannaneh Hajishirzi. 2019. Text Generation from Knowledge Graphs with Graph Transformers. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2284-2293, Minneapolis, Minnesota. Association for Computational Linguistics.
Wojciech Krysciński, Bryan McCann, Caiming Xiong, and Richard Socher. 2019. Evaluating the factual consistency of abstractive text summarization. arXiv, pages arXiv-1910.
Logan Lebanoff, Kaiqiang Song, and Fei Liu. 2018. Adapting the neural encoder-decoder framework from single to multi-document summarization. arXiv preprint arXiv:1808.06218.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461.
Yanpeng Li. 2018. Learning features from co-occurrences: A theoretical analysis. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2846-2854, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81.
Kathleen McKeown and Dragomir R Radev. 1995. Generating summaries of multiple news articles. In Proceedings of the 18th annual international ACM SIGIR conference on Research and development in information retrieval, pages 74-82.
Jonas Mueller, David Gifford, and Tommi Jaakkola. 2017. Sequence to better sequence: continuous revision of combinatorial structures. In International Conference on Machine Learning, pages 2536-2544.
Ratish Puduppully, Li Dong, and Mirella Lapata. 2019. Data-to-text generation with entity modeling. arXiv preprint arXiv:1906.03221.
Dragomir R. Radev and Kathleen R. McKeown. 1998. Generating natural language summaries from multiple on-line sources. Computational Linguistics, 24(3):469-500.
Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 379-389, Lisbon, Portugal. Association for Computational Linguistics.
Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073-1083, Vancouver, Canada. Association for Computational Linguistics.
Darsh J Shah, Tal Schuster, and Regina Barzilay. 2019. Automatic fact-guided sentence modification. arXiv preprint arXiv:1909.13838.
Darsh J Shah, Lili Yu, Tao Lei, and Regina Barzilay. 2021. Nutri-bullets: Summarizing health studies by composing segments. arXiv preprint arXiv:2103.11921.
Eva Sharma, Luyang Huang, Zhe Hu, and Lu Wang. 2019. An entity-driven framework for abstractive summarization. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3280-3291, Hong Kong, China. Association for Computational Linguistics.
Tianxiao Shen, Victor Quach, Regina Barzilay, and Tommi Jaakkola. 2020. Blank language models. arXiv preprint arXiv:2002.03079.
Mitchell Stern, William Chan, Jamie Kiros, and Jakob Uszkoreit. 2019. Insertion transformer: Flexible sequence generation via insertion operations. arXiv preprint arXiv:1902.03249.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008.
Wang Wenbo, Gao Yang, Huang Heyan, and Zhou Yuxiang. 2019. Concept pointer network for abstractive summarization. arXiv preprint arXiv:1910.08486.
Ronald J Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229-256.
Sam Wiseman, Stuart Shieber, and Alexander Rush. 2018. Learning neural templates for text generation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3174-3187, Brussels, Belgium. Association for Computational Linguistics.
Jie Yang and Yue Zhang. 2018. Ncrf++: An opensource neural sequence labeling toolkit. arXiv preprint arXiv:1806.05626.
Jianmin Zhang, Jiwei Tan, and Xiaojun Wan. 2018. Adapting neural single-document summarization model for abstractive multi-document summarization: A pilot study. In Proceedings of the 11th International Conference on Natural Language Generation, pages 381-390, Tilburg University, The Netherlands. Association for Computational Linguistics.