SlowGuess/NAACL_2022_OCR huggingface dataset:
Forrest Sheng Bao and Ge Luo and Hebi Li
Iowa State University, Ames, IA, USA
forrest.bao@gmail.com,{gluo,hebi}  $@$  iastate.edu
Minghui Qiu
minghui.qmh@alibaba-inc.com
Youbiao He
Ames, IA, USA
yh54@iastate.edu
Canonical automatic summary evaluation metrics, such as ROUGE, focus on lexical similarity which cannot well capture semantics nor linguistic quality and require a reference summary which is costly to obtain. Recently, there have been a growing number of efforts to alleviate either or both of the two drawbacks. In this paper, we present a proof-of-concept study to a weakly supervised summary evaluation approach without the presence of reference summaries. Massive data in existing summarization datasets are transformed for training by pairing documents with corrupted reference summaries. In cross-domain tests, our strategy outperforms baselines with promising improvements, and show a great advantage in gauging linguistic qualities over all metrics.
In natural language processing, the problem of summarization studies generating a summary from a source document which is longer than the summary. De facto metrics to judge a generated summary include ROUGE (Lin, 2004), BLEU (Papineni et al., 2002), and METEOR (Banerjee and Lavie, 2005). Previous work (Ng and Abrecht, 2015; Liu and Liu, 2008; Liu et al., 2016; Shang et al., 2018) agrees on two major drawbacks of them: 1) they favor lexical similarity, falling short on semantic similarity or linguistic quality, and 2) they require a reference summary which is often expensive to obtain (Zopf, 2018).
Initially, the first drawback is partially alleviated by replacing lexicons with their word embeddings (Ng and Abrecht, 2015; Ellouze et al., 2017; Ruseti et al., 2018; Xia et al., 2019). After the
1600 Amphitheater Parkway
yangyin7@gmail.com
Cen Chen
cecilia.cenchen@gmail.com
birth of transformers (Vaswani et al., 2017), this effort has expanded to sentence or document level, including reference-based (Zhang* et al., 2020; Zhao et al., 2019), and reference-free ones (Vasilyev et al., 2020; Scialom et al., 2019; Gao et al., 2020). The main difference between the two groups is whether a reference summary is needed when evaluating a machine-generated summary.
The two groups have complementary pros and cons. Reference-based metrics have a better performance, but they are impractical when summarization is used industrially, such as in customer support (Liu et al., 2019), team conversation (Zhang and Cranshaw, 2018), and bug reporting (Rastkar et al., 2014), where it is too costly to manually craft an equally massive amount of reference summaries. In contrast, without human written reference summaries, reference-free approaches generally perform poorer. Modern transformer-based reference-free approaches often rely on non-summarization tasks, such as QA (Vasilyev et al., 2020; Scialom et al., 2019). Such fact-focused approach makes them excel on content/fact aspects (still worse than reference-based ones) but not on linguistic ones. The non-summarization tasks also introduce noises.
Therefore, in this paper, as a proof of concept, we explore a hybrid or middle approach to combine the best of both worlds. Using document-summary pairs in existing summarization datasets, our weakly supervised approach mutates* reference summaries and pair them with documents to form training data and then use the trained model to evaluate unseen summaries in the presence of
*We avoid the term "augment" here because "augment" means making something better but what we are doing here is corrupting reference summaries.
Human Language Technologies, pages 2450 - 2458
documents without corresponding reference summaries. In this way, we make use of human written summaries, which are very precious, in training, but we do not need them in summary evaluation. We call our approach SueNes, which stands for "Summary evaluation by Negative sampling."
The quality of a summary is usually evaluated on two facets: content/fact aspects and linguistic qualities. Experiments later will show that a value of our approach is that we can use the same model architecture to build models that excel on different tasks by feeding training data from the same source but mutated in different strategies. For example, deleting words is the best for linguistic qualities while deleting sentences is the best for content/fact coverage.
Our approach is empirically compared against an array of existing metrics on three human summary evaluation datasets. Despite being training-based, our approach exhibits consistent results across various training domains which are all different from the test domain. It outperforms reference-free baselines with promising improvements on content/fact aspects, and further outperforms all existing metrics in gauging linguistic qualities.
In summary, our contributions or merits are:
- a simple but effective approach to reference-free† summary quality assessment,
- negative sampling for preparing training data from the unlabeled,
- one task/framework for multi-aspect judging,
- extensive cross-domain experiments to validate the effectiveness and domain robustness of our approach.
We hope our study can inspire more research into hybridizing reference-free and reference-based summary evaluation. Our code is at https://github.com/forrestbao/SueNes/
2 The Approach
A reference-free single-document summary quality assessor can be formulated as a regression function  $f(d, s) \in [0, 1]$  of an input document  $d = [t_1, t_2, \dots]$ , and a machine-generated summary  $s = [t_1', t_2', \dots]$ , where  $t_i$ 's and  $t_i'$ 's are text tokens. As a proof of concept, we explore an extremely lean
implementation of  $f$ : first  $d$  and  $s$  are jointly transformed into a vector representation  $\mathbf{e} = g(d, s)$ , and then it is mapped to a summary quality score via a fully-connected layer, i.e.,  $f(d, s) = \sigma(\mathbb{W}\mathbf{e})$ .
The function  $g$  can be implemented in the BERT (Devlin et al., 2019) style with an input sequence  $\left[\left[\mathrm{CLS}\right],t_1,t_2,\dots ,\left[\mathrm{SEP}\right],t_1',t_2',\dots ,\left[\mathrm{SEP}\right]\right]$ . The output on the [CLS] token is a joint representation of both the document  $d$  and the summary  $s$ .
While the human evaluation to a summary may cover multiple aspects, such as content/fact coverage and linguistics, a model of us will only yield one number. But by using different data mutation strategies, we can get models (different  $f$ 's) adept at different aspects of a summary.
2.2 Negative Sample Generation
It is impractical to train  $f$  with existing summarization datasets, such as CNN/DailyMail (Hermann et al., 2015; Nallapati et al., 2016), because they contain only high-quality, reference-class summaries written manually and thus are all of label 1. Some summary evaluation datasets, such as RealSumm (Bhandari et al., 2020), Newsroom (Grusky et al., 2018), and TAC2010 (NIST, 2010), do contain human ratings to system-generated summaries of various qualities. But they are too small, containing no more than 100 news articles or article groups each. Therefore, training against human ratings or in a supervised approach is impractical.
To work around, we propose a weakly supervised solution as depicted in Figure 1(a). Existing summarization datasets contain many document-summary pairs. For each pair  $\langle d,s\rangle$ , the reference summary  $s$  is mutated into  $K$  new summaries of different extents  $s_1,s_2,\dots ,s_K$ , which are then paired with the document to form new pairs
$$
\langle d, s _ {1} \rangle , \langle d, s _ {2} \rangle , \dots , \langle d, s _ {K} \rangle ,
$$
which are finally assigned targets to form the training data
$$
\left(\langle d, s _ {1} \rangle , y _ {1}\right), \left(\langle d, s _ {2} \rangle , y _ {2}\right), \dots , \left(\langle d, s _ {K} \rangle , y _ {K}\right).
$$
As illustrated in Figure 2, the training target  $y_{k\in [1..K]}$  is the percentage of intact content. For example, if  $30\%$  of tokens in a mutated summary are not original, then the label is 0.7. In addition, the original document-summary pair  $\langle d,s\rangle$  is also used in training with a target of 1.
The definition of "reference-free" is that reference summaries are not needed in the evaluation stage.
Figure 1: The weakly supervised training approach in this paper and the test of a trained model.
Mutations can happen at the token or sentence level, where tokens or sentences are randomly selected for mutation. A selected token or sentence is mutated in one of the three methods:
1. inserting a token/sentence from other summaries behind it,
2. deleting it, or
3. replacing it with a token/sentence from other summaries.
We do not mix different mutation levels nor mix different mutation methods when preparing the training data. Instead, our experiments study one combination of a mutation level and a mutation method, denoted as a mutation strategy, each time.
Figure 2: Training sample generation by mutation. Mutated text in dark blocks while intact text in gray blocks. Sizes are out of scale.
3.1 Test data
The ground truth of a summary's quality is human ratings to it. A model trained (Fig. 1(a)) is tested (Fig. 1(b)) against human ratings. Three test datasets are chosen below. Due to the limited number and sizes of human evaluation datasets, they are all in the news domain. The human evaluation protocols can be found in their respective references.
TAC2010 (NIST, 2010) is a multi-document (ten-document) summarization task reporting both factual and linguistic aspects. We use  $\sum_{i\in [1\dots 10]}f(d_i,s)$  to approximate the score of the summary  $s$  composed from ten documents  $d_{1}$  to  $d_{10}$ . We only use Set A of TAC2010 because Set B is not for regular summarization.
Newsroom (Grusky et al., 2018) also covers both factual (in INformativeness and RELevance) and linguistic (in COHerence and FLUency) aspects. For human ratings, three human annotators rate one pair of a document and machine-generated summary. The mean of their ratings on each aspect is used in our experiments.
RealSumm (Bhandari et al., 2020) focuses on only factual coverage. It covers 14 abstractive and 11 extractive summarizers published after 2018 and conducts human evaluation on the two groups separately.
Note that we do not and cannot train a model against the labels in a test set, as mentioned in  $\S 2.1$ .
2452 3
If a test set rates on multiple aspects, we do not train one model for each aspect. Nor do we train models for individual or a collection of test sets. We compute correlation between the predictions from our model and human ratings on each aspect of each test set.
3.2 Training data
Three widely used summarization datasets from three different domains are chosen for training: Billsum (Kornilova and Eidelman, 2019), Scientific-Papers/arXiv (Cohan et al., 2018), and Big-Patent (Sharma et al., 2019). Datasets from the news domain are avoided on purpose because the test data is in the news domain. This cross-domain setting allows us to examine whether a model is prone to domain differences. For each reference summary,  $K = 5$  mutated summaries are generated. The percentage of intact content is measured by the number of tokens and the number of characters for token-level and sentence-level mutations, respectively.
3.3 Baselines and upper bounds
To fairly compare, four recent metrics: BLANC (Vasilyev et al., 2020), SummaQA (Scialom et al., 2019), SUPERT (Gao et al., 2020) and LS-Score (Wu et al., 2020), are used as baselines because like our approach, they do not need a reference summary to judge a machine-generated summary, i.e., reference-free.
Human crafted reference summaries give reference-based metrics advantages. The results of reference-based metrics are included as soft upper bounds: ROUGE-1, ROUGE-2 and ROUGE-L (Lin, 2004), MoverScore (Zhao et al., 2019), BertScore (Zhang* et al., 2020), BLEU (Papineni et al., 2002), METEOR (Banerjee and Lavie, 2005), and  $\mathbf{S}^3$  (Peyrard et al., 2017).
3.4 Settings
Because the baselines use BERT, we use BERT as well for a fair comparison. Specifically, BERT-base uncased (L=12, H=768) is fine-tuned, with a learning rate of 1e-5, three epochs, and a batch size of 14. The input sequence is limited to 512 tokens using the round robin trimmer. The training loss is MSE as this problem is regression.
We use the summary-level (Peyrard et al., 2017) meta-evaluation strategy to report an approach's
average correlation with human ratings. Summary evaluation usually covers two types of aspects, contents/facts and linguistics. They are reported separately in Tables 1 and 2. Due to space limit, only the best mutation strategy is reported for each aspect group.
On content/fact aspects, the best mutation strategy is sentence deletion and our best models outperform baselines on all test datasets. Our approach makes the most improvement over baselines on RealSumm, a dataset much bigger than Newsroom and more modern than TAC2010, and the least improvement on TAC2010, the oldest dataset.
Table 1: Spearman's correlation on content/fact aspects.
Superscripts are ranks per aspect. Abs. and Ext. are two summarizer groups in RealSumm.
TAC2010 Pyramid	Newsroom INF REL	RealSumm Abs. Ext.
Our approach (mutated in sentence deletion)	Trained on:
Billsum	\( \mathbf{0.49}^{1} \)	\( \underline{\mathbf{0.70}}^{2} \)	\( \underline{\mathbf{0.61}}^{3} \)	0.26	0.01
arXiv	0.41	0.69	0.59	\( \mathbf{0.34}^{1} \)	\( \underline{\mathbf{0.12}}^{2} \)
BigPatent	0.42	\( \mathbf{0.75}^{1} \)	\( \mathbf{0.65}^{1} \)	\( \underline{\mathbf{0.33}}^{2} \)	\( \underline{\mathbf{0.13}}^{1} \)
Baselines	BLANC-tune	\( \underline{\mathbf{0.43}}^{3} \)	0.69	\( \underline{\mathbf{0.61}}^{2} \)	\( \underline{\mathbf{0.31}}^{3} \)	\( \underline{\mathbf{0.11}}^{3} \)
SummaQA-F1	0.30	0.57	0.52	0.22	0.08
SummaQA-CFD	0.29	0.54	0.44	0.24	0.05
SUPERT	\( \underline{\mathbf{0.48}}^{2} \)	\( \underline{\mathbf{0.69}}^{3} \)	0.60	0.25	0.07
LS-Score *	N/A	0.70	0.64	N/A	N/A
Upper bounds	R-1	0.56	0.32	0.28	0.63	0.22
R-2	0.64	0.15	0.13	0.56	0.22
R-L	0.50	0.30	0.26	0.60	0.21
MoverScore	0.72	0.22	0.22	0.50	0.19
BertScore	0.68	0.32	0.28	0.57	0.19
BLEU	0.60	-0.08	-0.01	0.30	0.16
METEOR	0.67	0.24	0.24	0.63	0.25
S3_pyr	0.73	0.27	0.25	0.64	0.24
S3_resp	0.73	0.25	0.22	0.63	0.24
Our best over baseline best (%)	2.71	8.67	6.40	9.72	16.42
Our average absolute deviation (%)	3.32	2.57	2.21	3.45	5.28
On linguistic aspects, the best mutation strategy is word deletion. Here, even our worst model cannot be outperformed by any baseline nor upper bound. As mentioned earlier, canonical metrics are lexical-based while modern reference-based and reference-free approaches focus on facts. Through mutating reference summaries, our approach can create summaries of different linguistic qualities. Although our approach makes big improvements over baselines on TAC2010 and Newsroom's FLUency, its edge is smaller on Newsroom's COHerence. A sentence-level scrambling mutation may improve our approach's performance on COHerence in the future.
* LS_Score results are only for Newsroom, which are copied from its paper, as we cannot run their code on other datasets after trying really hard. Several other researchers reported the same issue at https://github.com/whl97/LS-Score/issues. It is further excluded from the ranking because it is trained on the same domain as the test domain whereas all other baselines and our models are not.
Table 2: Spearman's correlation on linguistic aspects. Superscripts are ranks in each aspect/column.
TAC2010
Ling.	Newsroom
COH	FLU
Our approach (mutated in word deletion)	Trained on:
Billsum	0.461	0.652	0.652
arXiv	0.383	0.671	0.671
BigPatent	0.432	0.623	0.633
Baselines	BLANC-tune	0.29	0.59	0.53
SummaQA-F1	0.24	0.49	0.47
SummaQA-CFD	0.15	0.42	0.37
SUPERT	0.32	0.622	0.54
LS-Score*	N/A	0.63	0.59
Upper bounds	R-1	0.26	0.23	0.22
R-2	0.35	0.09	0.10
R-L	0.18	0.21	0.20
MoverScore	0.35	0.17	0.14
BertScore	0.36	0.27	0.24
BLEU	0.35	-0.06	-0.04
METEOR	0.34	0.17	0.17
S3_pyr	0.36	0.19	0.18
S3_resp	0.36	0.17	0.17
Our best over baseline best (%)	41.92	8.41	25.02
Our average absolute deviation (%)	2.72	1.71	1.74
3.6 Discussions
What is the best mutation? Across datasets, deletion-based mutations are most effective. The two kinds of deletions happen to be complementarily effective for two aspect groups: sentence deletion for content/fact aspects vs. word deletion for linguistic aspects. This is an advantage of our approach that under a uniformed framework, different summary quality aspects can be gauged by designing different mutation options.
The complementariness of sentence deletion and word deletion can be well explained as that removing a sentence from a reference summary reduces a great amount of key information while removing a word from a sentence changes it syntactically. We found that word-level mutations are less useful for content/fact aspects, probably because of the inertia of the context after words are altered.
Which training domain/dataset should I use? Due to the composition of summarizers and the limited data size in human evaluation, it is very hard to get a consistent ranking of metrics on different datasets (Bhandari et al., 2020). For example, in Table 1, Billsumm outperforms all baselines and its peers on TAC2010 but not the case on Newsroom and RealSumm.
Still, the impact of training domain seems manageable. The average absolute deviations across the training datasets/domains are given at the bottom of Tables 1 and 2. They mostly below  $3.5\%$ . A qualitative analysis shows that the variation seems more due to the characteristics of the text than
the domain. Legislative bills (Billsum) have lots of short, hierarchical clauses and thus differ from common English greatly. Scientific papers have many equations and cross-references. There are also many occurrences of Latex or MathML in the dataset arXiv. On top of that, all our experiments use different training and test domains. Hence we would say that the impact of domain variation is very small.
In this paper, we propose a weakly supervised approach to summary quality evaluation. A few mutation methods are introduced to make use of the massive, precious human written summaries in summarization datasets. In cross-domain experiments, our approach achieves better performance than baselines, especially on linguistic aspects. We hope this proof-of-concept study can inspire more reference-free summary evaluation.
Bao, Luo, Li, and He's work in this paper is partially supported by National Science Foundation (NSF) grants No. MCB-1821828 and No. CNS-1817089. The authors would also like to thank reviewers who have given precious feedback on improving this work.
Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65-72.
Manik Bhandari, Pranav Narayan Gour, Atabak Ashfaq, Pengfei Liu, and Graham Neubig. 2020. Reevaluating evaluation in text summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).
Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. 2018. A discourse-aware attention model for abstractive summarization of long documents. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers).
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of
2454 5
deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Samira Ellouze, Maher Jaoua, and Lamia Hadrich Belguith. 2017. Machine learning approach to evaluate multilingual summaries. In Proceedings of the Multi-Ling 2017 workshop on summarization and summary evaluation across source types and genres, pages 47-54.
Yang Gao, Wei Zhao, and Steffen Eger. 2020. SUPERT: Towards new frontiers in unsupervised evaluation metrics for multi-document summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1347-1354, Online. Association for Computational Linguistics.
Max Grusky, Mor Naaman, and Yoav Artzi. 2018. Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers).
Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In NeurIPS, pages 1693-1701.
Anastassia Kornilova and Vladimir Eidelman. 2019. *BillSum: A corpus for automatic summarization of US legislation*. In *Proceedings of the 2nd Workshop on New Frontiers in Summarization*, pages 48-56, Hong Kong, China. Association for Computational Linguistics.
Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Chia-Wei Liu, Ryan Lowe, Iulian Serban, Mike Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. How not to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2122-2132.
Chunyi Liu, Peng Wang, Jiang Xu, Zang Li, and Jieping Ye. 2019. Automatic dialogue summary generation for customer service. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '19, page 1957-1965, New York, NY, USA. Association for Computing Machinery.
Feifan Liu and Yang Liu. 2008. Correlation between rouge and human evaluation of extractive meeting