SlowGuess/NAACL_2021_OCR huggingface dataset:
Darsh J Shah $^{1}$  Lili Yu $^{2}$  Tao Lei $^{2}$  Regina Barzilay $^{1}$
<sup>1</sup>Computer Science and Artificial Intelligence Lab Massachusetts Institute of Technology
2ASAPP, Inc.
darsh@csail.mit.edu liliyu@asapp.com tao@asapp.com regina@csail.mit.edu
We present a method for generating comparative summaries that highlights similarities and contradictions in input documents. The key challenge in creating such summaries is the lack of large parallel training data required for training typical summarization systems. To this end, we introduce a hybrid generation approach inspired by traditional concept-to-text systems. To enable accurate comparison between different sources, the model first learns to extract pertinent relations from input documents. The content planning component uses deterministic operators to aggregate these relations after identifying a subset for inclusion into a summary. The surface realization component lexicalizes this information using a text-infilling language model. By separately modeling content selection and realization, we can effectively train them with limited annotations. We implemented and tested the model in the domain of nutrition and health – rife with inconsistencies. Compared to conventional methods, our framework leads to more faithful, relevant and aggregation-sensitive summarization – while being equally fluent.<sup>1</sup>
Articles written about the same topic rarely exhibit full agreement. To present an unbiased overview of such material, a summary has to identify points of consensus and highlight contradictions. For instance, in the healthcare domain, where studies often exhibit wide divergence of findings, such comparative summaries are generated by human experts for the benefit of the general public.² Ideally, this capacity will be automated given a large number of relevant articles and continuous influx of new ones that require a summary update to keep
Figure 1: We consider the database extracted from four Pubmed studies on Pears and Cancer. The key facts (bold) and consensus (contradiction) are realized in the text generated by our model.
it current. However, standard summarization architectures cannot be utilized for this task since the amount of comparative summaries is not sufficient for their training.
In this paper, we propose a novel approach to multi-document summarization based on a neural interpretation of traditional concept-to-text generation systems. Specifically, our work is inspired by the symbolic multi-document summarization system of (Radev and McKeown, 1998) which produces summaries that explicitly highlight agreements, contradictions and other relations across input documents. While their system was based on human-crafted templates and thus limited to a narrow domain, our approach learns different components of the generation pipeline from data.
To fully control generated content, we frame the task of comparative summarization as concept-to-text generation. As a pre-processing step, we ex
$^{1}$ Our code and data is available at https://github.com/darsh10/Nutribullets  
 $^{2}$ Examples include https://www.healthline.com and https://foodforbreastcancer.com.
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5213-5222 June 6-11, 2021. ©2021 Association for Computational Linguistics
tract pertinent entity pairs and relations (see Figure 1) from input documents. The Content Selection component identifies the key tuples to be presented in the final output and establishes their comparative relations (e.g., consensus) via aggregation operators. Finally, the surface realization component utilizes a text-infilling language model to translate these relations into a summary. Figure 1 exemplifies this pipeline, showing selected key pairs (marked in bold), their comparative relation - Contradiction (rows 1 &3 and rows 4&5 conflict), and the final summary. $^{3}$
This generation architecture supports refined control over the summary content, but at the same time does not require large amounts of parallel data for training. The latter is achieved by separately training content selection and content realization components. Since the content selection component operates over relational tuples, it can be robustly trained to identify salient relations utilizing limited parallel data. Aggregation operators are implemented using simple deterministic rules over the database where comparative relations between different rows are apparent. On the other hand, to achieve a fluent summary we have to train a language model on large amounts of data, but such data is readily available.
In addition to training benefits, this hybrid architecture enables human writers to explicitly guide content selection. This can be achieved by defining new aggregation operators and including new inference rules into the content selection component. Moreover, this architecture can flexibly support other summarization tasks, such as generation of updates when new information on the topic becomes available.
We apply our method for generating summaries of Pubmed publications on nutrition and health. Typically, a single topic in this domain is covered by multiple studies which often vary in their findings making it particularly appropriate for our model. We perform extensive automatic and human evaluation to compare our method against state-of-the-art summarization and text generation techniques. While seq2seq models receive competent fluency scores, our method performs stronger on task-specific metrics including relevance, content faithfulness and aggregation cognisance. Our method is able to produce summaries that receive
an absolute  $20\%$  more on aggregation cognisance, an absolute  $7\%$  more on content relevance and  $7\%$  on faithfulness to input documents than the next best baseline in traditional and update settings.
Text-to-text Summarization Neural sequence-to-sequence models (Rush et al., 2015; Cheng and Lapata, 2016; See et al., 2017) for document summarization have shown promise and have been adapted successfully for multi-document summarization (Zhang et al., 2018; Lebanonoff et al., 2018; Baumel et al., 2018; Amplayo and Lapata, 2019; Fabbri et al., 2019). Despite producing fluent text, these techniques may generate false information which is not faithful to the original inputs (Puduppully et al., 2019; Krysciński et al., 2019), especially in low resource scenarios. In this work, we are interested in producing faithful and fluent text cognizant of aggregation amongst input documents, where few parallel examples are available.
Recent language modeling approaches (Devlin et al., 2018; Stern et al., 2019; Shen et al., 2020; Donahue et al., 2020) can also be extended for text completion. Our work is a text-infilling language model where we generate words in place of relation specific blanks to produce a faithful summary.
Prior work (Mueller et al., 2017; Fan et al., 2017; Guu et al., 2018) on text generation also control aspects of the produced text, such as style and length. While these typically utilize tokens to control the modification, using prototypes to generate text is also very common (Guu et al., 2017; Li, 2018; Shah et al., 2019). In this work, we utilize aggregation specific prototypes to guide aggregation cognizant surface realization.
Data-to-text Summarization Traditional approaches for data-to-text generation have operated on symbolic data from databases. McKeown and Radev (1995); Radev and McKeown (1998); Barzilay et al. (1998) introduce two components of content selection and surface realization. Content selection identifies and aggregates key symbolic data from the database which can then be realized into text using templates. Unlike modern data-to-text systems (Wiseman et al., 2018; Puduppully et al., 2019; Sharma et al., 2019; Wenbo et al., 2019) these approaches capture document consensus and aggregation cognisance. While the neural approaches alleviate the need for human intervention, they do need an abundance of parallel data,
3We compare the selected content with other entries in the database, identifying two contradictions.
(c)
Figure 2: Illustrating the flow of our Nutribullets Hybrid system. In this example, our model takes in four Pubmed studies to produce a database (a). The Content Selection model selects two tuples (bold) and identifies the aggregation operator as Contradiction (b). Finally, the Surface Realization model takes in the tuples and aggregation operator to produce a summary which is faithful to input entities and aggregation cognizant (c).
which are typically from one source only. Hence, modern techniques do not deal with input documents' consensus in low resource settings.
Our goal is to generate a text summary  $y$  for a food from a pool of multiple scientific abstracts  $X$ . In this section, we describe the framework of our Nutribullets Hybrid system, illustrated in Figure 2.
We attain food health entity-entity relations, for both input documents  $X$  and the summary  $y$ , from entity extraction and relation classification modules trained on corresponding annotations (Table 2).
Notations: For  $N$  input documents, we collect  $X_{\mathcal{G}} = \{\mathcal{G}_p^x\}_{p = 1}^N$ , a database of entity-entity relations  $\mathcal{G}_p^x$ .  $\mathcal{G}_p = (e_1^k,e_2^k,r^k)_{k = 1}^K$  is a set of  $K$  tuples of two entities  $e_1$ ,  $e_2$  and their relation  $r$ .  $r$  represents relations such as the effect of a nutrition entity  $e_1$  on a condition  $e_2$  (see Table 2). We have raw text converted into symbolic data.
Similarly, we denote the corpus of summaries as  $Y = \{(y_{m},\mathcal{G}_{m}^{y},O_{m}^{y})_{m = 1}^{M}\}$ , where  $y_{m}$  is a concise summary,  $\mathcal{G}_m^y$  is the set of entity-entity relation tuples and  $O_{m}^{y}$  is the realized aggregation, in  $M$  data points.
Modeling: Joint learning of content selection, information aggregation and text generation for multi-
document summarization can be challenging. This is further exacerbated in our technical domain with few parallel examples and varied consensus amongst input documents. To this end, we propose a solution using Content Selection and Aggregation and Surface Realization models.
Raw text from  $N$  input documents is converted into a mini-database  $X_{\mathcal{G}}$  of relation tuples. The content selection and aggregation model operates on such symbolic data. We use  $X_{\mathcal{G}}$  and  $Y$  to train the content selection model. During inference, we identify from  $X_{\mathcal{G}}$  a subset  $C$  of content to present in the final output. In order to produce a summary cognizant of consensus amongst inputs, we identify the aggregation operator  $O$  based on  $C$  and other relevant tuples in  $X_{\mathcal{G}}$ .
The surface realization model produces a relevant, faithful and aggregation cognizant output. The model is trained only using  $Y$ . During inference, the model realizes text using the selected content  $C$  and the aggregation operator  $O$ .
3.2 Content Selection and Aggregation
Our content selection model takes a mini-database of entity-entity relation tuples  $X_{\mathcal{G}}$  as input, and outputs the key tuples  $C$  and the aggregation operator  $O$ .
Content selection and aggregation consists of two parts - (i) identifying key content  $P(C|X_{\mathcal{G}})$  and (ii) subsequently identifying the aggregation operator  $O$  using  $C, X_{\mathcal{G}}$ .
Content Selection Identifying key content in
4We train an entity tagger and relation classifier to predict  $\mathcal{G}$  and also for computing knowledge based evaluation scores. More details on models and results are shared later.
Aggregation Operator	Deterministic Rule
Under-Reported Population Scoping Contradiction Agreement	|Pubmed Studies| < Threshold |Specific Population| < Threshold (ema=ea&&ema==2&&rm!=rn) for any two tuples m,n from different studies None of the Above
Table 1: Deterministic Rules to identify the Aggregation Operator.
volves selecting important, diverse and representative tuples from a database. While clustering and selecting from the database tuples is a possible solution, we model our content selection as a finite Markov decision process (MDP). This allows for an exploration of different tuple combinations while incorporating delayed feedback from various critical sources of supervision (similarity with target tuples, diversity amongst selected tuples etc). We consider a multi-objective reinforcement learning algorithm (Williams, 1992) to train the model. Our rewards (Eq. 2) allow for the selection of informative and diverse relation tuples.
The MDP's state is represented as  $s_t = (t, \{c_1, \ldots, c_t\}, \{z_1, z_2, \ldots, z_{m-t}\})$  where  $t$  is the current step,  $\{c_1, \ldots, c_t\}$  is the content selected so far and  $\{z_1, z_2, \ldots, z_{m-t}\}$  is the remaining entity-entity relation tuples in the  $m$ -sized database. The action space is all the remaining tuples plus one special token,  $Z \cup \{STOP\}$ . The number of actions is equal to  $|m-t| + 1$ . As the number of actions is variable yet finite, we parameterize the policy  $\pi_\theta(a|s_t)$  with a model  $f$  which maps each action and state  $(a, s_t)$  to a score, in turn allowing a probability distribution over all possible actions using softmax. At each step, the probability that the policy selects  $z_i$  as a candidate is:
$$
\pi_ {\theta} (a = z _ {i} | s _ {t}) = \frac {\exp (f (t , \hat {z} _ {i} , c _ {i} ^ {*}))}{\sum_ {j = 1} ^ {m - t + 1} \exp (f (t , \hat {z} _ {j} , c _ {j} ^ {*}))} (1)
$$
where  $c_{i} * = \arg \max_{c_{j}} (\cos(\hat{z}_{i}, \hat{c}_{j}))$  is the selected content closest to  $z_{i}$ ,  $\hat{z}_{i}$  and  $c_{i} *$  are the encoded dense vectors,  $\cos(u, v) = \frac{u \cdot v}{||u|| \cdot ||v||}$  is the cosine similarity of two vectors and  $f$  is a feedforward neural network with non-linear activation functions that outputs a scalar score for each action  $a$ .
The selection process starts with  $Z$ . Our module iteratively samples actions from  $\pi_{\theta}(a|s_t)$  until selecting STOP, ending with selected content  $C$  and a corresponding reward. We can even allow for the selection of partitioned tuple sets by adding
an extra action of "NEW LIST", which allows the model to include subsequent tuples in a new group.
We consider the following individual rewards:
-  $\mathcal{R}_e = \sum_{c\in C}\cos (e_{1c},e_{1y}) + \cos (e_{2c},e_{2y})$  is the cosine similarity of the structures of the selected content  $C$  with the structures present in the summary  $y$  (each summary structure accounted with only one  $c$ ), encouraging the model to select relevant content.
-  $\mathcal{R}_d = \mathbb{1}[\max_{i,j}(\cos (\hat{c}_j,\hat{c}_i)) < \delta ]$  computes the similarity between pairs within selected content  $C$ , encouraging the selection of diverse tuples.
-  $r_p$  is a small penalty for each action step to encourage concise selection.
The multi-objective reward is computed as
$$
\mathcal {R} = w _ {e} \mathcal {R} _ {e} + w _ {d} \mathcal {R} _ {d} - | C | r _ {p}, \qquad (2)
$$
where  $w_{e}, w_{d}$  and  $r_{p}$  are hyper-parameters.
During training the model is updated based on the rewards. During inference the model selects an ordered set of key and diverse relation tuples corresponding to appropriate health conditions.
Consensus Aggregation Identifying the consensus amongst the input documents is critical in our multi-document summarization task. We model the aggregation operator of our Content Selection using simple one line deterministic rules as shown in Table 1. The rules are applied to the key  $C$  entity-entity relation pairs in context of  $X_{\mathcal{G}}$ . In our example in Figure 1,  $O$  is Contradiction because of rows 1&3 and rows 4&5 (rows 1&3 only would also make it Contradiction).
3.3 Surface Realization
The surface realization model  $P(y|O,C)$ , performs the critical task of generating a summary guided by both the entity-entity relation tuples  $C$  and the aggregation operator  $O$ . The model allows for robust, diverse and faithful summarization compared to traditional template and modern seq2seq approaches.
5 STOP and NEW LIST get special embeddings.
Relation Type	e1	e2	r	Example
Causing	Food, Nutrition	Condition	Increase, Decrease, Satisfy, Control, Unclear/Insignificant	(tart cherry juice, melatonin levels, increase), (water, daily fluid needs, satisfy)
Containing	Food, Nutrition	Nutrition	Contain	(blueberries, antioxidants, contain)
Table 2: Details of entity-entity relationships that we study and some examples of  $\left( {{e}_{1},{e}_{2},r}\right)$
We propose to model this process as a prototypedriven text infilling task. The entities from  $C$  are used as fixed tokens with relations as special blanks in between these entities. This is prefixed by a prototype summary corresponding to  $O$ . For the example shown in Figure 2, we concatenate using  $|SEN|$  a randomly sampled contradictory summary "Kale contains substances ... help fight cancer ... but the human evidence is mixed." to  $C$ "<blank> pears <controls> ovarian cancer <decreases> breast cancer <blank)". The infilling language model produces text corresponding to relations between entities while maintaining an overall structure which is cognizant of  $O$ .
The model is trained on the few sample summaries from the training set using  $\mathcal{G}_m^y$  and  $O_{m}^{y}$  to produce  $y_{m}$ . Providing aggregation and content guidance during generation alleviates the low-resource issue.
4 Summary and Update Setting
In this section we describe the setting of summary updates. In a real world setting, we would often receive new input documents such as scientific studies about the same subject which necessitate a change in an old summary.
In context of our food and health summarization task, the goal is to update an old summary about a food and health condition on receiving results from new scientific studies from Pubmed. Our model can accommodate this scenario fairly easily. We describe the minor changes to the Content Selection and Aggregation and Surface Realization models for such a setting.
We are provided an original summary and can extract it's content  $C'$  and can also construct the mini-database  $X_{\mathcal{G}}$  from the text of the new documents. We identify the aggregation between the new studies'  $X_{\mathcal{G}}$  and original summary's content  $C'$  first. Depending on the aggregation identified,
corresponding content  $C$  is selected from  $X_{\mathcal{G}}$ . For instance, in case of a contradiction, we are keen on identifying content leading to this contradiction. The subsequent Surface Realization is dependent on  $O$ , the selected  $C$  and the  $C^{\prime}$  present in the original summary  $(P(y|O,C + C^{\prime}))$ .
Dataset We utilize a real world dataset for Food and Health summaries, crawled from https:// www.healthline.com/nutrition (Shah et al., 2021). The HealthLine dataset consists of scientific abstracts as inputs and human written summaries as outputs. The dataset consists of 6640 scientific abstracts from Pubmed, each averaging 327 words. The studies in these abstracts are cited by domain experts when writing summaries in the Healthline dataset, forming natural pairings of parallel data. Individual summaries average 24.5 words and are created using an average of 3 Pubmed abstracts. Each food has multiple bullet summaries, where each bullet typically talks about a different health impact (hydration, diabetes etc). We assign each food article randomly into one of the train, development or test splits. Entity tagging and relation classification annotations are provided for the Pubmed abstracts and the healthline summaries.
Settings: We consider three settings.
1. Single Issue: We use the individual food and health issue summaries as a unique instance of food and single issue setting. We split 1894 instances  $80\%$ ,  $10\%$ ,  $10\%$  to train, dev and test.
2. Multiple Issues: We group each food's article Pubmed abstract inputs and multiple summary outputs as a single parallel instance. 464 instances are split  $80\%$ ,  $10\%$ ,  $10\%$  to train, dev and test.
3. Summary Update: We consider two kinds of updates – new information is fused to an existing summary and new information contradicts an existing summary. For fusion we consider single issue summaries that have multiple conditions from different Pubmed studies (bananas + low blood pressure from one study and bananas + heart health from another study). We partition the Pubmed
<sup>6</sup>Summaries in our training data are labelled with  $O_{m}^{y}$  as belonging to one of the four categories of Under-reported, Population Scoping, Contradiction or Agreement to accommodate such training.
MODEL	Automatic Evaluation	Human Scores
ROUGEL	KG(G)	KG(I)	AG	RELEVANCE	FLUENCY
Copy-gen	0.12	0.21	0.50	0.64	1.93	1.89
GraphWriter	0.14	0.03	0.69	0.64	1.86	2.76
Entity Data2text	0.16	0.13	0.57	0.67	2.03	3.43
Transformer	0.20	0.21	0.64	0.67	2.66	3.76
Ours	0.18	0.30	0.76	0.89	3.03	3.46
Table 3: Automatic evaluation – Rouge-L score (RougeL), KG in gold(G), KG in input(I) and Aggregation Cognisance (Ag) in our model and various baselines in the single issue setting, is reported. Human evaluation on Relevance and Fluency, on 1-4 Likert scale from 3 annotators, is also reported. The best results are in bold.
studies to stimulate an update. The contradictory update setting is where we artificially introduce conflicting results in the input document set so that the aggregation changes from Agreement to Contradictory. We have a total of 103 test instances. All models are trained atop of Single issue data.
Evaluation We evaluate our systems using the following automatic metrics. Rouge is an automatic metric used to compare the model output with the gold reference (Lin, 2004).  $KG(G)$  computes the number of entity-entity pairs with a relation in the gold reference, that are generated in the output. This captures relevance in context of the reference.  $KG(I)$ , similarly, computes the number of entity-entity pairs in the output that are present in the input scientific abstracts. This measures faithfulness with respect to the input documents. Aggregation Cognisance ( $Ag$ ) measures the accuracy of the model in producing outputs which are cognizant of the right aggregation from the input, (Under-reported, Contradiction or Agreement). We use a rule-based classifier to identify the aggregation implied by the model output and compare it to the actual aggregation operator based on the input Pubmed studies.
In addition to automatic evaluation, we have human annotators score our models on relevance and fluency. Given a reference summary, relevance indicates if the generated text shares similar information. Fluency represents if the generated text is grammatically correct and written in well-formed English. Annotators rate relevance and fluency on a 1-4 likert scale (Albaum, 1997). We have 3 annotators score every data point and report the average across the scores.
Baselines In order to demonstrate the effectiveness of our method, we compare it against text2text and
data2text state-of-the-art (sota) methods.
Copy-gen (Text2text): See et al. (2017) is a sota technique for summarization, which can copy from the input or generate words.
Transformer (Text2text): Hoang et al. (2019) is a summarization system using a pretrained Transformer.
GraphWriter (Data2text): Koncel-Kedziorski et al. (2019) is a graph transformer based model, which generates text using a seed title and a knowledge graph. Takes the database  $X_{\mathcal{G}}$  as input.
Entity (Data2text): Puduppully et al. (2019) is an entity based data2text model, takes  $X_{\mathcal{G}}$  as input.
Implementation Details Our policy network is a three layer feedforward neural network. We use a Transformer (Vaswani et al., 2017) implementation for Surface Realization. We train an off-the-shelf Neural CRF tagger (Yang and Zhang, 2018) for entity extraction. We use BERT (Devlin et al., 2018) based classifiers to predict the relation between two entities in a text trained using crowdsourced annotations from (Shah et al., 2021). Further implementation details can be found in A.