SlowGuess/NAACL_2022_OCR huggingface dataset:
We use ROUGE-scores (Lin, 2004) for our evaluation, in which ROUGE-1 is the representative score. ROUGE-2 and ROUGE-L are reported in the Appendix. We also report the number of interactions to show that our novel sampling techniques can significantly speed up the agent's learning.
7 Results and Discussion
7.1 Reward model study
We first observe the efficacy of the proposed reward model in the summarization process. To do that, we compare our reward model to the reward models of APRIL (Gao et al., 2018) and OpenAI Ziegler et al. (2019). For the preference prediction task, the reward scores assigned to candidate summaries are used to determine the preferred summary. In general, the preferred summary is the one with a higher score. We select a subset of documents from the datasets to train the reward model. Each document can be used to construct three triples corresponding to the three objectives mentioned above. The number of pretraining/interactive training/testing documents is 1000/1000/1000 for Billsum and Livedoor, and 2000/2000/2000 for Reddit TIFU, respectively.
(a) Average testing accuracy of APRIL, OpenAI and ROMSR when using all three sets.
(b) Active learning on RedditTIFU dataset of ROMSR when using all three sets and set 3 only
Figure 2: Reward model: (a) Accuracy of 3 models on all sets (b) The ROUGE-1 score of our ROMSR.
Figure 2 (a) demonstrates that our reward model (ROMSR) significantly outperforms other methods when training on all three objectives. APRIL uses a linear regression model combined with a heuristics function that is tuned for the quality objective. This specific design limits the model to handle all three objectives. OpenAI's results exhibit slow learning progress because it is a large neural network using
joint features from a pair of texts, while our reward model learns to use respective-order on a feature space of a single document. Moreover, the reconstruction loss from the autoencoder helps the model to avoid overfitting to small training samples.
To verify the necessity of using three objectives for training the reward model, we compare two versions of our reward model: one is trained with all sets of objectives and another with the third objective (the traditional quality objective). Two reward models are used in active learning to finetune the backbone. Figure 2(b) shows that training on all three objectives converges faster.
7.2 Fine-tuning backbone model using human preferences
This section shows our comprehensive experiments on three datasets and three scenarios to prove the effectiveness of our human-preference-guided learning with RL, and our LRS and DSS sampling.
Active learning We compare our sampling techniques with the baselines, including random offline sampling and methods without offline sampling, such as the standard baseline and APRIL. Figure 3 shows that combining online data with randomly selected samples harms the agent during online fine-tuning. Meanwhile, selecting prior low-reward samples and similar documents boosts accuracy faster than not using offline data. After 64 interactions, ROUGE-1 reaches almost the highest. To validate the actual impact of ROMSR, we also test the reward model in the RL pipeline compared to the APRIL's reward. Due to the OpenAI reward is inferior to the APRIL's reward, we ignore the OpenAI's reward in this experiment. The results in Figure 3 show that APRIL (orange) is inferior to our ROMSR (blue) in all datasets.
Few-shot learning Figure 4 describes the quality of the models: LRS, DSS, Random, and the standard baseline during interactions. The standard baseline without offline sampling can not improve the summarization agent with a few interactions and eventually worsen the agent. In contrast, LRS and DSS can significantly improve the model after 4 interactions. Naively using the offline data still harms the model, similar to active learning.
Online learning Due to computation limitations, we only examine this setting on the Reddit TIFU dataset. Also, prior experiments demonstrated that LRS and DSS are equally good sampling strategies.
As DSS runs much faster than LRS, we choose DSS as our representative method in this experiment. We report the performance of DSS, Random, and the standard baseline in Figure 5.
The results show that random sampling is again inferior to other methods in this online scenario. The standard baseline shows good performance yet becomes unstable and drops performance later. Our DSS demonstrates fast and stable convergence, consistently achieving the highest ROUGE-1 score throughout interactions.
7.3 Ablation study and model analysis
Impact of top- $k$  selected offline documents To investigate the effect of  $k$ -the number of sampled offline documents, we try different  $k$  values ( $k = 1$  and  $k = 4$ ) on the RedditTIFU dataset, and the results are depicted in Figure 6a. As observed,  $k = 4$  is slightly better, but  $k = 1$  runs and converges faster. Hence, we use  $k = 1$  in all experiments.
Random sampling makes low rewards We investigate the failure of random sampling. We keep track of the reward assigned to randomly selected offline documents across interactions. Figure 6b shows that, in general, DSS and LRS give better samples, indicated by higher assigned rewards, than random sampling. DSS and LRS's rewards are also much more consistent, showing less variance than random sampling. Thus, offline documents from random sampling provide little benefit and make the training unstable and unreliable.
We also analyze the distribution characteristics of selected offline documents in Figure 6c. The green line represents the distribution of similarity scores between online and random offline samples. The red one is the distribution of similarity scores between online and DSS samples. It is noticeable that the similarity score of DSS is in the range from 0.0 to 0.15, whereas this score of Random is up to 0.5. It is expected that sampling offline data will create a coherent training distribution for online learning, but Random skews the training distribution. Therefore, Random harms the performance.
Running time We assess the running time of our proposed method compared to the baseline on Reddit TIFU and BillSum in Table 1. The assessment is conducted on a single Tesla T4. On Reddit TIFU, our DSS takes nearly 3 hours (64 interactions) to reach almost the highest ROUGE-1 score of 28.6, while the baseline reaches only 26.0 of ROUGE-1 after 64 interactions. The baseline takes only
Figure 3: Active learning: ROUGE-1 with the mean and standard deviation over 5 runs.
Figure 4: Fewshot learning: ROUGE-1 with the mean and standard deviation over 5 runs.
Figure 5: Online learning on Reddit TIFU: ROUGE-1 with the mean and standard deviation over 5 runs.
30m faster than the DSS. In light of BillSum, the DSS is slower than the baseline, about 50 minutes for 96 interactions. However, the DSS is better than the baseline in terms of ROUGE-1. Despite that our proposed method takes more computing resources than the baseline, our approach requires much fewer interactions, saving human resources in the real HITL setting. In terms of real-time interactive feedback, it takes around 3 minutes for training in each interaction on the Reddit dataset, and approximately 1 minute for training in each
Method	Interactions	Run time	ROUGE-1
The Reddit TIFU dataset
Baseline	64	2h20m	26.0
DSS	64	2h50m	28.6
The BillSum dataset
Baseline	96	50m	29.4
DSS	96	1h40m	30.3
Table 1: The running time of active learning with offline sampling.
interaction in the BillSum dataset, which is not really real-time. It is noted that the amount of time to provide feedback is not included because we use simulated oracles. Hence, in actual applications, we can fine-tune the model while the oracle gives feedback because it takes time for the oracle to read the summary and decide their preference. We believe such a parallel process will make the methods feasible to use in real-time settings.
This work proposes a novel approach for learning reward functions in preference learning. By utilizing offline data with a reward model that focuses on the distance between the summary and the document, our method improves the ROUGE-scores
(a) Effect of  $k$  to the performance on RedditTIFU dataset
(b) Reward during training
Figure 6: Ablation study: (a) Performance with different  $k$  values; (b) Quality of selected samples; (c) Semantic similarity between online documents and offline documents.
(c) Similarity distribution between online samples and selected offline samples
by  $2 - 5\%$  in comparison to the APRIL framework, while beating the random baseline by a large margin on three different datasets. Our experimental results also suggest that by applying low-reward sampling or document-similarity sampling, we can achieve efficiency in terms of both running time and the number of human interactions. Regarding limitations, our method is not tested with large online stream data, which may cause catastrophic forgetting. Future work will confirm our model's effectiveness for abstractive summarization.
We would like to thank anonymous ACL ARR reviewers and Senior Area Chairs who gave constructive comments for our paper.
Ralph Allan Bradley and Milton E Terry. 1952. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-345.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Yang Gao, Christian M. Meyer, and Iryna Gurevych. 2018. APRIL: Actively learning to summarise by combining active preference learning and reinforcement learning. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4120-4130, Brussels, Belgium. Association for Computational Linguistics.
Byeongchang Kim, Hyunwoo Kim, and Gunhee Kim. 2019. Abstractive summarization of Reddit posts with multi-level memory networks. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2519-2531, Minneapolis, Minnesota. Association for Computational Linguistics.
Tomonori Kodaira and Mamoru Komachi. 2018. The rule of three: Abstractive text summarization in three bullet points. In Proceedings of the 32nd Pacific Asia Conference on Language, Information and Computation, Hong Kong. Association for Computational Linguistics.
Anastassia Kornilova and Vladimir Eidelman. 2019. *BillSum: A corpus for automatic summarization of US legislation*. In *Proceedings of the 2nd Workshop on New Frontiers in Summarization*, pages 48-56, Hong Kong, China. Association for Computational Linguistics.
Siyao Li, Deren Lei, Pengda Qin, and William Yang Wang. 2019. Deep reinforcement learning with distributional semantic rewards for abstractive summarization. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6038-6044, Hong Kong, China. Association for Computational Linguistics.
Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Yang Liu and Mirella Lapata. 2019. Text summarization with pretrained encoders. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3730-3740, Hong Kong, China. Association for Computational Linguistics.
David Maxwell, Leif Azzopardi, and Yashar Moshfeghi. 2017. A study of snippet length and informativeness: Behaviour, performance and user experience. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 135-144.
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Ranking sentences for extractive summarization with reinforcement learning. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1747-1759, New Orleans, Louisiana. Association for Computational Linguistics.
Duy-Hung Nguyen, Bao-Sinh Nguyen, Nguyen Viet Dung Nghiem, Dung Tien Le, Mim Amina Khatun, Minh-Tien Nguyen, and Hung Le. 2021. Robust deep reinforcement learning for extractive legal summarization. In International Conference on Neural Information Processing, pages 597-604. Springer.
Ramakanth Pasunuru and Mohit Bansal. 2018. Multi-reward reinforced summarization with saliency and entailment. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 646-653, New Orleans, Louisiana. Association for Computational Linguistics.
Romain Paulus, Caiming Xiong, and Richard Socher. 2018. A deep reinforced model for abstractive summarization. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net.
Avinesh P.V.S and Christian M. Meyer. 2017. Joint optimization of user-desired content in multi-document summaries by learning from user feedback. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1353-1363, Vancouver, Canada. Association for Computational Linguistics.
Nils Reimers and Iryna Gurevych. 2019. SentenceBERT: Sentence embeddings using Siamese BERTnetworks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982-3992, Hong Kong, China. Association for Computational Linguistics.
John Schulman, Philipp Moritz, Sergey Levine, Michael I. Jordan, and Pieter Abbeel. 2016. High-dimensional continuous control using generalized advantage estimation. In 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. ArXiv preprint, abs/1707.06347.
Prafull Sharma and Yingbo Li. 2019. Self-supervised contextual keyword and keyphrase retrieval with self-labelling.
Aditya Siddhant and Zachary C. Lipton. 2018. Deep Bayesian active learning for natural language processing: Results of a large-scale empirical study. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2904-2909, Brussels, Belgium. Association for Computational Linguistics.
Nisan Stiannon, Long Ouyang, Jeff Wu, Daniel M Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. 2020. Learning to summarize from human feedback. ArXiv preprint, abs/2009.01325.
Victor Uc-Cetina, Nicolas Navarro-Guerrero, Anabel Martin-Gonzalez, Cornelius Weber, and Stefan Wermter. 2021. Survey on reinforcement learning for language processing.
Christian Wirth, Riad Akrour, Gerhard Neumann, and Johannes Furnkranz. 2017. A survey of preference-based reinforcement learning methods. Journal of Machine Learning Research, 18(136).
Xingjiao Wu, Luwei Xiao, Yixuan Sun, Junhang Zhang, Tianlong Ma, and Liang He. 2021. A survey of human-in-the-loop for machine learning.
Baosheng Yu, Tongliang Liu, Mingming Gong, Changxing Ding, and Dacheng Tao. 2018. Correcting the triplet selection bias for triplet loss. In Proceedings of the European Conference on Computer Vision (ECCV), pages 71-87.
Fabio Massimo Zanzotto. 2019. Human-in-the-loop artificial intelligence. Journal of Artificial Intelligence Research, 64:243-252.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2019. Fine-tuning language models from human preferences. *ArXiv preprint*, abs/1909.08593.
A Hyper-parameters and Implementation details
Due to the limitation of resources, our backbone bases on BERT-base with 12 Transformer blocks with the hidden size is 768, and 110M parameters. We train our model with the learning rate of 1e-5. During online fine-tuning with RL, we set  $k$  equals the number of online documents, where  $k$  is the number of selected offline data. The noisy preference probability  $nc$  is 0.1. For evaluation, we use ROUGE-score with parameters  $-c95 - m - r1000 - n2$ .
In Figure 8, we show ROUGE-2 and ROUGE-L scores with the same setting as section 7.2. The results of few-shot and online learning setting is depicted in Figure 9 and 7 respectively.
Figure 7: Online learning on RedditTIFU dataset: ROUGE-2 and ROUGE-L score with the mean and std over 5 runs
Figure 8: Active learning: ROUGE-2 and ROUGE-L score with the mean and std over 5 runs
Figure 9: Fewshot learning: ROUGE-2 and ROUGE-L score with the mean and std over 5 runs