SlowGuess/NAACL_2022_OCR huggingface dataset:
$$
L _ {m l} = - \sum_ {t = 1} ^ {N} \log p \left(y _ {t} ^ {*} \mid y _ {1} ^ {*}, \dots , y _ {t - 1} ^ {*}, x\right) \tag {1}
$$
For our RL optimization, we use self-critical policy gradient training as in Paulus et al. (2018); Rennie et al. (2017). At each time-step, we produce an output  $y^{s}$  by sampling from the current decoding probability,  $p(y_{t}^{s}|y_{1}^{s},\dots,y_{t - 1}^{s},x)$ , as well as an output  $\hat{y}$  obtained by greedily decoding from the current probability distribution. We define a reward function  $r(y,x,y^{*})\in [0,1]$ , i.e., the reward function compares  $y$  with  $x$  and  $y^{*}$ . The RL loss function  $L_{rl}(x,y^{*}) =$ :
$$
\left(r \left(\hat {y}, x, y ^ {*}\right) - r \left(y ^ {s}, x, y ^ {*}\right)\right) \sum_ {t = 1} ^ {N} \log p \left(y _ {t} ^ {s} \mid y _ {1} ^ {s}, \dots , y _ {t - 1} ^ {s}, x\right) \tag {2}
$$
As in Paulus et al. (2018) and Pasunuru and Bansal (2018), we use a mixture of the two losses above:
$$
L _ {\text {m i x e d}} = \gamma_ {r l} L _ {r l} + \gamma_ {m l} L _ {m l}, \tag {3}
$$
where  $\gamma_{rl}$  and  $\gamma_{ml}$  are tunable hyperparameters used as scaling factors. Rather than applying weights to each reward, we follow Pasunuru and Bansal (2018) and optimize  $L_{mixed}$  by alternating rewards in each minibatch.
5.2 Rewards
We use the following RL reward functions: (1) textual entailment (NLI) for faithfulness, and (2) semantic area to measure the coverage of a summary in a semantic space.
NLI for Faithful Summarization: We use the degree of entailment of summaries given input answers as a reward to promote faithfulness of answer
summulation. Falke et al. (2019) define NLI as a measure of faithfulness for ranking summaries as follows: Let  $\mathcal{N}$  be an NLI model which, given a claim  $c$  and a premise  $p$ , computes  $\mathcal{N}(p,c)$ , the probability that the claim is entailed by the premise. We use this to calculate the NLI score for a summary  $y$  consisting of  $N_{s}$  sentences:
$$
\operatorname {N L I} (y, x) = \frac {1}{N _ {s}} \sum_ {i = 1} ^ {N _ {s}} \max  _ {s \in x} \mathcal {N} (s, y _ {i _ {s}}) \tag {4}
$$
Semantic Area for Multi-Perspective Summarization: We aim to reward summaries that include more of the perspectives found in the input answers. To achieve diverse extractive summarization, Yogatama et al. (2015) embed sentences in the semantic space and then select those sentences whose convex hull maximizes the volume in that space. This idea of semantic volume is also used to measure the semantic overlap between summaries and references in Jung et al. (2019). We use semantic volume as a proxy for covering multiple perspectives; the summary with the larger semantic volume covers a wider range of views discussed in the input. We make use of sentence-transformers (Reimers and Gurevych, 2019b) to obtain sentence embeddings for each sentence. We project each embedding onto two dimensions using Principal Component Analysis (PCA) as in Jung et al. (2019), and thus, our volume calculation reduces to an area calculation, which we refer to as Semantic Area. We use min-max normalization to keep the reward between 0 and 1. We split the dataset into training, validation, and testing sets of size 3131, 500, and 1000 examples. For relevance labeling, we train RoBERTa (Liu et al., 2019) for binary relevance classification with the user question and sentence as inputs. We train with a polynomial decay learning rate scheduler with learning rate  $2\mathrm{e} - 5$ , using the Adam optimizer (Kingma and Ba, 2015) for three epochs. We compare this model to one trained on the ANTIQUE (Hashemi et al., 2020) relevance data for query-sentence relevance. The data consists of Yahoo! answers and relevance labels on a scale from 1-4, with 1-2 not relevant and 3-4 relevant.
For experiments in ClusterSumm and E2ESumm, our baseline abstractive text summarization model is BART (Lewis et al., 2020), a pretrained denoising autoencoder that builds off of the sequence-to-sequence transformer of Vaswani et al. (2017). For E2ESumm results, our primary focus, we also ap
True Rel	True Not Rel
Predicted Rel	4324	3349
Predicted Not Rel	5664	25088
Table 5: RoBERTa confusion matrix on SentSelect.
ply several state-of-the-art abstractive summarization models such as T5-base (Raffel et al., 2019). For the cluster summarization task, the input is the individual sentences clustered by the annotators, while for the cluster fusion step, the input is the concatenation of the cluster summaries. For E2ESumm, input to the models is the question concatenated with input answers. For both summarization tasks, we fine-tune BART using a polynomial decay learning rate scheduler with learning rate  $3 \times 10^{-5}$ , using the Adam optimizer (Kingma and Ba, 2015). We train with 500 warm-up steps and 20,000 total steps and pick the model with the best label-smoothed cross-entropy (Szegedy et al., 2016) validation loss. T5 is trained for 3 epochs with a linear learning rate scheduler. In RL experiments, we train using BART from scratch, as opposed to using a model already fine-tuned on answer summarization, as we found that this model better learned to follow the given rewards. Following similar ratios in Lu et al. (2019), we set  $(\gamma_{rl}, \gamma_{ml}) = (0.9, 0.1)$ . Hyperparameters are tuned on the validation set.
6 Results & Discussion
We provide strong baseline results for the SentS-elect, ClusterSumm, ClusterSummFusion, and E2ESumm subtasks of AnswerSumm as a basis for future work.
The best results for SentSelect are yielded by RoBERTa relevance classification as illustrated in Table 5. RoBERTa yields an F1 score of 0.49. Despite being the highest, the relatively low result points to the difficulty and subjectivity of selecting relevant sentences for community question answering fora. This is further supported by the observed low IAA of fair agreement (Fleiss Kappa of 0.25). Moreover, concatenating the sentences labeled as relevant on the test set as a final summary results in long summaries with high recall (82.81 ROUGE-1 Recall). This suggests that much of the important information to be summarized can be captured by this relevance model. The ANTIQUE-trained model obtains an F1 score of 0.41 and notably predicts many false positives (71%). While this
Task	ROUGE-1/2/L
ClusterSumm	30.98/10.61/26.22
ClusterSummFusion	51.64/32.67/47.13
Table 6: ROUGE scores for ClustSumm and Fusion summarization tasks, showing ClustSumm as one of the bottlenecks in E2ESumm performance.
Model	ROUGE-1/2/L
BART-large (Lewis et al., 2020)	28.17/8.61/24.01
BART-large-aug	29.10/9.15/24.63
T5-base (Raffel et al., 2019)	25.10/6.58/21.30
BART-rel-oracle	30.98/10.61/26.22
model performs worse on this relevance classification task, we find that using this trained model for automatically generated data allows for an improved downstream summarization model, when compared to the better classifier trained solely on our manually-annotated data. Accordingly, we opt for using the ANTIQUE-trained model in our overall summarization task. The improved performance is likely due to more sentences being labeled as relevant (implicitly encoding a recall bias), allowing for more sentences to be sent to the clustering algorithm, a noisy step itself, ensuring better quality clusters.
Results for ClusterSumm and ClusterSummFusion are shown in Table 6. These results point to the difficulty of ClusterSumm as one of the sources of difficulty for E2ESumm performance, as ClusterSummFusion can be done fairly easily. We believe that some difficulties found in ClusterSumm are also found in the E2ESumm task.
The results for E2ESumm are presented in Table 7. BART-large outperforms T5 model, but scores are rather low when compared to the extractive oracle above. To investigate this further, we train a BART-only model using the question concatenated with the oracle relevant sentences chosen by the annotators, BART-rel-oracle. BART-rel-oracle significantly outperforms the vanilla model. This suggests that improved content selection would boost performance. However, we believe that the primary cause of the low performance is the difficulty in learning the compression rate and abstractiveness of the gold summaries. The percentage of novel uni-grams in BART is only  $4\%$ , as opposed to the  $21\%$  present in the gold summaries. This suggests that despite being trained on more abstractive data, BART is not learning (not gen
Table 7: Model comparison for E2ESumm.
Task	ROUGE-1/2/L	NLI	Semantic Area
BART	28.17/8.61/24.01	0.74	0.04
BART-aug	29.10/9.15/24.63	0.77	0.01
BART-aug + RL	28.81/8.96/24.72	0.76	0.05
Table 8: A comparison of model ROUGE, NLI, and Semantic Area scores.
eralizing) how to abstract well enough. We also note the model trained on additional augmented data through our automatic pipeline, BART-aug, achieves a large performance boost compared to vanilla BART, thereby validating the efficacy of our automatic pipeline for potential applications to new domains. It should be noted that we experimented with augmenting our manually-curated data with data from CQASumm, but performance did not improve over vanilla BART. Hence, the task is indeed sensitive to the quality and type of data used for augmentation.
The results of adding RL rewards to BART trained with data augmentation are shown in Table 8. Both BART with augmented data and RL rewards achieve higher NLI scores than the baseline, while only the model with RL rewards obtains a higher Semantic Area score. The improved ROUGE score for BART-aug likely results from training on additional data that resembles the target domain, as in Fabbri et al. (2021), while noise in the unsupervised data may reduce the Semantic Area scores. The addition of RL rewards improves the semantic area score over the augmented model, although the slight decrease in ROUGE-1/2 show that semantic area does not completely align with ROUGE score. We analyzed 25 model outputs for factual consistency and found that the models are largely factually consistent, and very extractive (BART-aug + RL having the fewest novel unigrams at  $3.9$ ). This suggests these differences in NLI score do not exhibit a large qualitative difference in faithfulness, and the lower NLI score of the RL model may be from the introduction of the semantic area reward. Also, we note that the gold summaries themselves have low NLI and Semantic Area scores of 0.46 and 0.03. As the gold summaries are more abstractive, the entailment relationship between them and the input may not be as straightforward as the primarily extractive model outputs. This phenomenon suggests the need for improved metrics and rewards for abstractive factual consistency and semantic coverage. We provide example summaries in the supplementary materials.
We develop an annotation pipeline for multiperspective answer summarization, introducing the largest human-annotated dataset for this task. We benchmark state-of-the-art models on the content selection, cluster summarization, and end-to-end summarization subtasks of this dataset. We also introduce a pipeline for answer summarization data augmentation that boosts summarization performance. Through an analysis of the effects of reinforcement-learning rewards and qualitative examination of model outputs, we point to difficulties in these tasks and areas for future improvement in content selection, abstraction levels, and metrics for model comparison.
As we propose a novel conversation summarization dataset creation pipeline and modeling components, this section is divided into the following two parts.
8.1 New Dataset
Intellectual Properties and Privacy Rights We make use of publicly-available StackExchange data for all our annotations. We manually reviewed our dataset output for quality and potential problems.
Compensation for Annotators Compensation was determined by standard in-house rates, amounting to about $6 per data point collected.
8.2 NLP Application
Bias Biases may exist in the datasets, such as political bias and gender bias in Yahoo! Answers. Thus, models trained on these datasets may propagate these biases.
Misuse Potential and Failure Mode When used as intended, applying the summarization models described in this paper can save people much time. However, the current models are still prone to producing hallucinated summaries, and in such a case, they may contribute to misinformation on the internet. We move the needle in faithful summarization in this paper, but further research is needed to ensure the faithfulness of abstractive summaries to address this issue, as this issue is present among all current abstractive summarization models.
Environmental Cost The experiments described in the paper make use of V100 GPUs. We used up to 8 GPUs per experiment. The experiments
may take several hours. Several dozen experiments were run due to parameter search, and future work should experiment with distilled models for more light-weight training. We note that while our work required extensive experiments to draw sound conclusions, future work will be able to draw on these insights and need not run as many large-scale comparisons. Models in production may be trained once for use using the most promising settings.
Wen Chan, Xiangdong Zhou, Wei Wang, and Tat-Seng Chua. 2012. Community answer summarization for multi-sentence question with group L1 regularization. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 582-591, Jeju Island, Korea. Association for Computational Linguistics.
Tanya Chowdhury and Tanmoy Chakraborty. 2019. Cqasumm: Building references for community question answering summarization corpora. In Proceedings of the ACM India Joint International Conference on Data Science and Management of Data, pages 18-26.
Tanya Chowdhury, Sachin Kumar, and Tanmoy Chakraborty. 2020. Neural abstractive summarization with structural attention. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020, pages 3716-3722. ijcai.org.
Yang Deng, Wai Lam, Yuexiang Xie, Daoyuan Chen, Yaliang Li, Min Yang, and Ying Shen. 2020a. Joint learning of answer selection and answer summary generation in community question answering. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 7651-7658. AAAI Press.
Yang Deng, Wenxuan Zhang, Yaliang Li, Min Yang, Wai Lam, and Ying Shen. 2020b. Bridging hierarchical and sequential context modeling for question-driven extractive answer summarization. pages 1693-1696.
Ori Ernst, Ori Shapira, Ramakanth Pasunuru, Michael Lepioshkin, Jacob Goldberger, Mohit Bansal, and Ido Dagan. 2020. Superpal: Supervised proposition alignment for multi-document summarization and derivative sub-tasks. ArXiv preprint, abs/2009.00590.
Alexander Fabbri, Simeng Han, Haoyuan Li, Haoran Li, Marjan Ghazvininejad, Shafiq Joty, Dragomir Radev, and Yashar Mehdad. 2021. Improving zero
and few-shot abstractive summarization with intermediate fine-tuning and data augmentation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 704-717, Online. Association for Computational Linguistics.
Alexander Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir Radev. 2019. Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1074-1084, Florence, Italy. Association for Computational Linguistics.
Tobias Falke, Leonardo F. R. Ribeiro, Prasetya Ajie Utama, Ido Dagan, and Iryna Gurevych. 2019. Ranking generated summaries by correctness: An interesting but challenging application for natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2214-2220, Florence, Italy. Association for Computational Linguistics.
Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. 2019. ELI5: Long form question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3558-3567, Florence, Italy. Association for Computational Linguistics.
Demian Gholipour Ghalandari, Chris Hokamp, Nghia The Pham, John Glover, and Georgiana Ifrim. 2020. A large-scale multi-document summarization dataset from the Wikipedia current events portal. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1302-1308, Online. Association for Computational Linguistics.
Xiaotao Gu, Yuning Mao, Jiawei Han, Jialu Liu, You Wu, Cong Yu, Daniel Finnie, Hongkun Yu, Jiaqi Zhai, and Nicholas Zukoski. 2020. Generating representative headlines for news stories. In WWW '20: The Web Conference 2020, Taipei, Taiwan, April 20-24, 2020, pages 1773-1784. ACM / IW3C2.
Amir Hadifar, Lucas Sterckx, Thomas Demeester, and Chris Develder. 2019. A self-training approach for short text clustering. In Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP-2019), pages 194–199, Florence, Italy. Association for Computational Linguistics.
Helia Hashemi, Mohammad Aliannejadi, Hamed Zamani, and W Bruce Croft. 2020. Antique: A non-factoid question answering benchmark. In European Conference on Information Retrieval, pages 166-173. Springer.
Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. 2020. spaCy: Industrial-strength Natural Language Processing in Python.
Taehee Jung, Dongyeop Kang, Lucas Mentch, and Eduard Hovy. 2019. Earlier isn't always better: Subaspect analysis on corpus and system biases in summarization. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3324-3335, Hong Kong, China. Association for Computational Linguistics.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pretraining for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computational Linguistics.
Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Peter J. Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. 2018. Generating wikipedia by summarizing long sequences. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. ArXiv preprint, abs/1907.11692.
Yuanjie Liu, Shasha Li, Yunbo Cao, Chin-Yew Lin, Dingyi Han, and Yong Yu. 2008. Understanding and summarizing answers in community-based question answering services. In Proceedings of the 22nd International Conference on Computational Linguistics (Coling 2008), pages 497-504, Manchester, UK. Coling 2008 Organizing Committee.
Yao Lu, Linqing Liu, Zhile Jiang, Min Yang, and Randy Goebel. 2019. A multi-task learning framework for abstractive text summarization. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 - February 1, 2019, pages 9987-9988. AAAI Press.
Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Caglar Gulçehre, and Bing Xiang. 2016. Abstractive text summarization using sequence-to-sequence RNNs and beyond. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pages 280-290, Berlin, Germany. Association for Computational Linguistics.
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1797-1807, Brussels, Belgium. Association for Computational Linguistics.
Vinay Pande, Tanmoy Mukherjee, and Vasudeva Varma. 2013. Summarizing answers for community question answer services. In *Language Processing and Knowledge in the Web*, pages 151-161. Springer.
Ramakanth Pasunuru and Mohit Bansal. 2018. Multireward reinforced summarization with saliency and entailment. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 646-653, New Orleans, Louisiana. Association for Computational Linguistics.
Ramakanth Pasunuru, Asli Celikyilmaz, Michel Galley, Chenyan Xiong, Yizhe Zhang, Mohit Bansal, and Jianfeng Gao. 2021. Data augmentation for abstractive query-focused multi-document summarization. In AAAI.
Romain Paulus, Caiming Xiong, and Richard Socher. 2018. A deep reinforced model for abstractive summarization. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. *ArXiv* preprint, abs/1910.10683.
Md Rashadul Hasan Rakib, Norbert Zeh, Magdalena Jankowska, and Evangelos Milios. 2020. Enhancement of short text clustering by iterative classification. In International Conference on Applications of Natural Language to Information Systems, pages 105-117. Springer.
Marc'Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. 2016. Sequence level training with recurrent neural networks. In 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings.