SlowGuess/NAACL_2022_OCR huggingface dataset:
Duy-Hung Nguyen $^{1,\ast}$ , Nguyen Viet Dung Nghiem $^{1,\ast}$ , Bao-Sinh Nguyen $^{1}$ , Dung Tien Le $^{1}$ , Shahab Sabahi $^{1}$ , Minh-Tien Nguyen $^{1,2}$ , Hung Le $^{3}$
$^{1}$ Cinnamon AI, 10th floor, Geleximco building, 36 Hoang Cau, Dong Da, Hanoi, Vietnam. {hector, henry93, simon, nathan, sshahab, ryan.nguyen} @cinnamon.is  $^{2}$ Hung Yen University of Technology and Education, Hung Yen, Vietnam.
tiennm@utehy.edu.vn  
<sup>3</sup>Deakin University, Australia.  
thai.le@deakin.edu.au
For summarization, human preferences is critical to tame outputs of the summarizer in favor of human interests, as ground-truth summaries are scarce and ambiguous. Practical settings require dynamic exchanges between humans and AI agents wherein feedback is provided in an online manner, a few at a time. In this paper, we introduce a new framework to train summarization models with preference feedback interactively. By properly leveraging offline data and a novel reward model, we improve the performance regarding ROUGE scores and sample-efficiency. Our experiments on three various datasets confirm the benefit of the proposed framework in active, few-shot and online settings of preference learning.
The advent of AI has changed business practices, though the human involvement is still important. The human roles in interaction with AI-powered machines have been evolving under the concept of human-in-the-loop (HITL) (Zanzotto, 2019). HITL allows humans to actively participate in supervising AI systems by approving, rejecting, or re-labeling current outputs, and providing expert-guided advices to the system. It will also act as the unique source of external knowledge from humans. By observing the outputs of AI systems, humans can hand-pick some potential outcomes and then feedback to the models for better performance.
In NLP, document summarization is considered as a subjective task (Stiennon et al., 2020). They argued that it would be hard to quantify what makes a "good summary" without the human judgment input. Collecting human feedback and evaluating the crafted summaries from documents for building the training datasets is time-consuming (Wu et al.,
2021) and labor-expensive. It is true particularly where the domain knowledge is required. Moreover, Uc-Cetina et al. (2021) argued the importance of the user's intentions for modeling Natural Language Understanding with high performance.
Given these attributions, we are more interested in deploying an interactive HITL-based text summarization framework, which continuously collects the user-feedback to consequently improve model prediction robustness. Here, the user's intention is implicitly acknowledged as a factor influencing the extraction of important sentences from the source documents. Upon this formulation, the AI model will be trained with human-produced summaries and adapted as more human-feedback is fed in.
Previous studies used human feedback to rank the label of objects (Wirth et al., 2017), which employed reinforcement learning to minimize the user's effort to provide feedback for training a ranker. However, these approaches rank the entire solution space including relevant and irrelevant pairs, which are a waste of computing power. To tackle this shortcoming, Siddhant and Lipton (2018) employed Bayesian Optimization, which substitutes the standard uncertainty-based acquisition functions for active learning. However, the model still consumes a lot of computing power for a larger number of iterations, and is vulnerable to the curse of dimensionality of input data. Recently, researchers proposed to learn a reward model simulating human preferences (Ziegler et al., 2019). The reward model was then used to translate real-time human feedback to the reward score for fine-tuning the model under RL training. The method was designed for online learning of language models and required numerous interactions to achieve good performance. However, how to achieve sample-efficiency is still an open question.
In this paper, we propose a novel interactive pref
Findings of the Association for Computational Linguistics: NAACL 2022, pages 1919 - 1930 July 10-15, 2022 ©2022 Association for Computational Linguistics
erence learning for the summarization task. To do that, we fine-tune a pretrained extractive summarizer as the backbone with reinforcement learning by using a reward model that enforces the distance-based order of preferences. The reward model is trained to differentiate two summaries regarding the topic, length, and quality (human preferences). To enable sample-efficiency, we propose to utilize offline data, which was previously used to pretrain the model. We show that naively using the offline data is harmful for preference learning. Instead, we introduce two mechanisms to selectively sample offline data in favor of human feedback learning. Our sampling strategies focus on low-rewarded samples or documents which are similar to fine-tuning data. We demonstrate that our method can be used in various settings: active, few-shot and online learning. Tested on three summarization datasets, our method consistently achieves significantly better results compared to competitive baselines in each setting. In summary, our contribution is three-fold:
- We propose a new RL-based preference learning system for the summarization task by using a novel reward model.
- We propose sampling mechanisms to efficiently leverage offline data for preference learning of extractive summarization.
- We conduct extensive empirical studies on three summarization datasets, showing that our proposed method outperforms competitive baselines in various settings such as active, online, and few-shot learning scenarios.
2.1 Summarization with RL
Direct reward The most direct way to have a reward function in reinforcement learning for summarization is to match the candidate summary to the reference (gold) summary (Narayan et al., 2018; Paulus et al., 2018). Put in the HITL setting, humans are required to provide the gold summary for training the machine, which is prohibitively expensive and can be ambiguous. Thus, we focus on rewards constructed from preference feedback, in which humans only need to indicate the better summary between two candidate summaries.
Preference reward As stated in the introduction, the preference from humans is much more accessible and consistent. In this approach, the frame
works (Gao et al., 2018; Stiennon et al., 2020; Nguyen et al., 2021) consist of two main steps: 1) Preference learning that gives a score, which mimics human evaluation, to a summary of a document. 2) Reinforcement learning based on the reward model. These works have not examined the summarization problem in an interactive training scheme, which will be addressed in this paper.
2.2 Preference learning in NLP
Preference learning aims at obtaining the ranking (i.e. total ordering) of objects from pairwise preferences, in which the linear Bradley-Terry (BT) model (Bradley and Terry, 1952) is one of the most studied methods. Later, the APRIL framework (Gao et al., 2018) shows that it can reduce the number of required comparisons by using active selection with uncertainty sampling (P.V.S and Meyer, 2017). On the other hand, the OpenAI framework (Stiennon et al., 2020) uses a neural structure to predict the score given a document and its summary. Its advantage is the learning with direct human evaluation (score for each summary) or human preferences (comparison between 2 summaries). However, the input space is the product of two documents, so it needs much more human feedback to achieve good performance. In contrast, our proposed reward model treats the input as only one document, and thus is simpler and has fewer parameters in the learning process. That is more suitable for the interactive learning context where the number of training samples is limited.
Reinforcement learning is a popular and effective approach to utilize human preferences. The APRIL framework (Gao et al., 2018) trains a summarizer by using a pretrained preference model. The work of Ziegler et al. (2019) employs preference-based RL to fine-tune a deep language model for sentiment classification and document summarization tasks. They only rely on online data during fine-tuning. We instead make use of offline data to accelerate the learning process.
3.1 The backbone model
We use BERTSUM (Liu and Lapata, 2019) as the backbone for extractive summarization. Given a document with a set of sentences, the model uses BERT (Devlin et al., 2019) for learning hidden vectors of sentences by using the modified [CLS] token of each sentence. The vectors are fed into
an inter-sentence layer by using the Transformer for learning the inter-relationship among sentences. Important sentences are extracted by using a sigmoid function for sentence importance estimation.
3.2 Fine-tuning with reinforcement learning
Proximal policy optimization (PPO) To finetune the backbone with interactive feedback, we treat the summarization process as a sequential decision-making process so that we can employ RL (Stiannon et al., 2020; Ziegler et al., 2019). The RL agent traverses all sentences in the original document, and at each time step, it classifies the current sentence  $s_i$  into two labels: important ( $y_i = 1$ ) and unimportant ( $y_i = 0$ ). We consider the backbone model trained with supervised learning as the initial policy. The policy is then optimized by using PPO (Schulman et al., 2017) as our RL method. The objective of training PPO is:
$$
\begin{array}{l} J ^ {P P O} (\theta) = \mathbb {E} [ \min  (r a (\theta) \hat {A} _ {\theta_ {o l d}} (s _ {i}, y _ {i}), \\ \mathrm {c l i p} (r a (\theta), 1 - \epsilon , 1 + \epsilon) \hat {A} _ {\theta_ {o l d}} (s _ {i}, y _ {i})) ] \\ \end{array}
$$
where  $\theta$  is the current policy's parameters,  $\hat{A}_{\theta_{old}}(s_i,y_i)$  is the advantage calculated at the old policy parameters  $\theta_{old}$  before each updated policy iteration, by using any advantage estimation algorithm to transform the rewards (Schulman et al., 2016), and  $ra(\theta) = \frac{\pi_{\theta}(y_i|s_i)}{\pi_{\theta_{old}}(y_i|s_i)}$  is the ratio between the new policy and the old policy. This is the idea of importance sampling that evaluates the new policy with samples collected from the older policy. If the ratio  $ra$  falls outside the range  $1 - \epsilon$  and  $1 + \epsilon$ , the advantage function will be clipped. PPO uses the objective to avoid big changes between new and old policies.
Reward schemes Besides the final reward that evaluates the quality of the whole summary, we follow Pasunuru and Bansal (2018); Li et al. (2019) to use an additional reward to constrain policy updates. Let  $\pi_{\theta_0}$  denotes the supervised trained backbone model and  $\pi_{\theta}$  is the one that we optimize with RL. The reward at the time step  $i$  is:
$$
r _ {i} = - \beta_ {K L} \log [ \frac {\pi_ {\theta} (y _ {i} | s _ {i} , D)}{\pi_ {\theta_ {0}} (y _ {i} | s _ {i} , D)} ] + \mathbb {1} \times (i = n) \times r ^ {M}
$$
where  $\beta_{KL}$  is the KL coefficient,  $n$  is the final time step corresponding to the total number of sentences in the document  $D$ . For the intermediate time step  $i$  ( $i < n$ ), the reward is the negative KL
The online data is queried in three scenarios: active learning, few-shot learning, and online learning
Figure 1: The overview of our framework. In which the backbone is in charge of generating two summaries for a document. Then the oracle selects which summary is better for a given document. The reward model afterward transforms the oracle's preference into a discrete signal to optimize the backbone. Our framework contains two novel components: efficient sampling from offline data and the preference-guided reward model.
divergence between the output distribution of the backbone model and the current policy. For the final time step  $i$  ( $i = n$ ), when the model obtains the complete summary  $S$ , the reward model takes the summary and the original document to produce the final reward  $r^{M}$  (see Section 5).
4 Interactive Learning with Online RL
In our problem, a backbone summarizer and a reward model are pretrained with offline data by using supervised learning. We aim to update the models to adapt with a stream of online data by using RL. Given a document  $D$  from the online data, the backbone (Section 3.1) extracts two summaries for an oracle (i.e., a humans) to provide preference feedback. Each time the backbone receives feedback in one interaction. After receiving feedback from the oracle, the reward model is fine-tuned by using supervised learning (Section 5). The feedback is then translated to a reward via a reward model, which is later used to fine-tune the backbone by using RL (Section 3.2), thereby improving the summary quality for future interactions.
This interactive mechanism is applicable to three scenarios: active, online, and few-shot learning (Section 4.2). Offline data can be employed during online fine-tuning to speed up the learning process by using the selective sampling method (Section 4.3). The common objective of all scenarios is to maximize summarization quality while minimizing the number of interactions. We describe the overview of our framework in Figure 1.
4.2 Interaction plots
Active learning Given a pool of unlabeled samples, the agent queries the most informative samples at each iteration to create summaries. Then the agent asks the oracle to select which summary is better for the queried documents. The agent learns from the feedback of the oracle subsequently. This setting has been studied by Gao et al. (2018).
Online learning Different from active learning, for online learning, each unlabeled instance is typically drawn one at a time from the data stream, continuously being sent to the system. The agent processes the data and is not allowed to choose the document. Ziegler et al. (2019) is the early work that promotes the online learning scenario.
Few-shot learning Similar to the online setting, the system passively receives samples from the data stream. This time, it is only given a few unlabeled samples (up to 4 documents) and manages to learn from the data by continuously producing summaries and receiving feedback. To our best knowledge, this is the first time that the few-shot setting is examined in preference learning.
4.3 Efficient interaction from offline data
To achieve sample efficiency in terms of the number of interactions between the backbone and the oracle, we argue that it is critical to leverage prior data used in the pretraining phase. We observe that the naive use of prior data by using random sampling is inefficient. Hence, we propose two novel offline-data sampling methods: low-reward and document-similarity. We describe all sampling mechanisms as follows.
Random sampling (Random) We randomly select  $k$  offline documents and combine them with online data to fine-tune the backbone model.
Low-reward sampling (LRS) Intuitively, high-quality summaries should be given a higher reward by the reward model. Samples that have extracted summaries with low rewards have not been well-learned by the summarizer. As the result, we select top  $k$  documents having the lowest rewards from the reward model.
Document-similarity sampling (DSS) Conventionally, the backbone outputs good summaries when the training and testing distributions are similar. Therefore, we sample offline documents that share similar semantics with online documents. To
do that, we encode the documents by using BERT (Devlin et al., 2019), and then compute their similarity by using the Cosine distance.  $k$  offline documents have the lowest distance to the online counterparts are selected for online training.
5 The Reward Model for Preference Learning
5.1 Training procedure
We construct a reward model to generate preference rewards for finetuning the backbone summarizer with RL. The reward model should simulate human preferences, capable of assigning a higher reward to the preferred summary. For preference learning, we use the relative order between two summary candidates to train the reward model. The correct order is given by humans via their preferences.
For traditional learning, the model has access to training documents of the dataset. Each document  $D$  has a gold summary  $S_{g}$ , which is assumed to be preferred to other candidate summaries (silver or machine-generated  $S_{m}$ ), denoted by  $S_{g} \succ S_{m}$ . The training data is in the form of a triplet  $(D, S_{g}, S_{m})$  where  $S_{g} \succ S_{m}$ . We argue that this is insufficient to achieve a good reward model. From our observation, before reaching the quality of gold summaries, the machine often produces off-topic and too-short answers. In addition, the work of Maxwell et al. (2017) also suggested that the oracle prefers longer summaries because they feel that longer summaries are more readable, clear, and informative. Thus, we introduce three objectives as follows.
The topic objective Aiming to detect the document's topic:  $(D, S_{g}, S_{m})$ . We generate the training set of the topic objective as follows. For each document  $D$  that has a gold summary  $S_{g}$ , we select a summary  $S_{g}^{\prime}$  from a different document  $D^{\prime}$  (randomly chosen) and expect that  $S_{g} \succ S_{g}^{\prime}$ . If we miss the human summary, we could use machine-generated summaries instead  $((D, S_{m}, S_{m}^{\prime})$  such that  $S_{m} \succ S_{m}^{\prime}$ ).
The length objective Aiming to detect the amount of summary information (i.e. the length of the summary). For each document  $D$ , we used a pretrained summarization model to generate a long  $(S_{ml})$  and a short  $(S_{ms})$  summary to obtain the triplet  $(D, S_{ml}, S_{ms})$ . The correct order is  $S_{ml} \succ S_{ms}$ .
The quality objective Aiming to detect the general quality of a summary. For each document  $D$ , we generate a summary  $S_{m}$  which has a similar length to the human summary and include the gold summary to compose the triplet for training  $(D, S_{g}, S_{m})$ . We note that the data of this objective is similar to the traditional data used in Stiennon et al. (2020), which needs humans to provide a gold-silver summary or correct order between arbitrary summary candidates. On the contrary, the data of topic and length objectives can be created programmatically and thus save human effort.
5.2 Respective-order mapping
For summarization, we argue that the reward model's interpretation of ordering should be determined based on the distance between the summary and document representations. Therefore, we aim to learn global representing mapping  $\phi$  such that:
$$
S _ {1} \succ S _ {2} \iff d (\phi (D), \phi (S _ {1})) <   d (\phi (D), \phi (S _ {2}))
$$
where  $d()$  is the Euclidean distance,  $\phi(D)$  is the representation of the document  $D$  in the respective-order mapping. Then, the reward for a summary  $S$  is computed as follows.
$$
r ^ {M} (D, S) = \frac {s c o r e (\phi (D) , \phi (S)) - s c o r e _ {m i n}}{s c o r e _ {m a x} - s c o r e _ {m i n}}
$$
where  $score(D, S) = \frac{1}{1 + \exp(d(\phi(D), \phi(S)))}$  is the unnormalized score between  $D$  and  $S$ ;  $score_{min}$  and  $score_{max}$  are the minimum and maximum score over all pairs of document-summary, respectively. We name our reward model as ROMSR (Respective-Order Mapping Score Reward).
5.3 Representation learning
To train  $\phi$ , first, we embed all documents that appear in triplet sets. To obtain important features of a document, we use the joint-embedding of a Transformer (capture a general meaning of the document) (Reimers and Gurevych, 2019; Devlin et al., 2019) and keyword feature vectors (capture important keywords in the document) (Sharma and Li, 2019). We denote this transformation as  $f(D)$ .
In fact, this feature vector  $f(D)$  can be used to compare the similarity between two documents. However, it is not robust enough to satisfy the relative comparison from the triplets. Therefore, we learn a new embedding  $(encode(f(D)))$  that tries to follow all the triplet conditions. In order to be a good representation of  $f(D)$ , the embedding is
also reinforced by the reconstruction loss of the autoencoder. The embedded representation is our target mapping  $\phi(D) = \text{encode}(f(D))$ . The loss for the learning of the document representation  $\mathcal{L}$  is the combination of the autoencoder reconstruction loss  $\mathcal{L}_{AE}$  and the respective-order loss  $\mathcal{L}_{RO}$  (the triplet loss) as follows.
$$
\mathcal {L} = \mathcal {L} _ {A E} + \mathcal {L} _ {R O}
$$
$$
\mathcal {L} _ {A E} = \sum_ {D \in \mathbb {D} \cup \mathbb {S}} \| f (D) - d e c o d e (\phi (D)) \|
$$
$$
\begin{array}{l} \mathcal {L} _ {R O} = \sum_ {(D, S, S ^ {\prime}) \in T} \max  (0, d (\phi (D), \phi (S)) \\ - d (\phi (D), \phi (S ^ {\prime})) + \alpha) \\ \end{array}
$$
where  $\mathbb{D},\mathbb{S}$  is the set of original/summarized documents,  $T$  is the union of all triplet sets, and  $\alpha$  is the margin hyperparameter controlling the stretch in the representation space (Yu et al., 2018).
We use three benchmark summarization datasets for our evaluation. BillSum comprises 22,218 US bill and human-written summary pairs collected from the US Government Publishing Office (Kornilova and Eidelman, 2019). The data is split into 18,949 training and 3,269 testing bills. Reddit TIFU is an English dataset collected from Reddit. It contains 12,000 posts divided into TIFU-long and TIFU-short (documents have less than 400 words) (Kodaira and Komachi, 2018). Livedoor contains Japanese articles crawled from the Livedoor News website. Each article consists of three summary sentences written by editors (Kim et al., 2019).
Our experiments use the training set to pretrain the backbone and reward models. Then, we randomly select 5000 samples from the original training data to create offline data. The online data is created from the original validation data. For active learning and online learning settings, we sample up to 320 documents from the online data. For few-shot learning, only 4 documents are randomly selected as online data. We aim to minimize the number of interactions to reach the highest performance on the whole online dataset.
6.2 Simulated interactions
Given one original document with two corresponding generated summaries and a standard metric ROUGE score, the oracle knows the ground-truth summary, and prefers the summary that has the higher ROUGE score w.r.t the ground-truth. Theoretically, the ideal data for preference-based interactive learning is consistent preferences in which a higher-scored summary is always selected. However, we believe that perfect selection is impossible for real-world applications because humans can occasionally misinterpret the intention when the presented candidates have similar qualities. Therefore, in this work, we consider noisy preferences with uniform probability  $nc \in [0,1]$ , that randomly selects which summary is better.
6.3 Baselines
We construct a standard model (baseline) that just finetunes the backbone summarizer (see Section 3.1) with online data, coupled with our proposed reward model and PPO training. The combination of BERT and PPO is shown effectively in the offline document summarization task (Nguyen et al., 2021). The baseline is similar to that of Ziegler et al. (2019), which uses human preferences and PPO to finetune language models. However, the latter uses a different reward model.
To show the efficacy of our reward model, we compare ours with the reward model of Ziegler et al. (2019) (OpenAI) and the uncertainty sampling reward model in APRIL (Gao et al., 2018) in the preference prediction task (see Section 7.1).
For few-shot and online learning, we compare our methods with the standard baseline. For active learning, we also compare our methods with APRIL. For all settings, we build a baseline that randomly samples from offline data to compare against our proposed sampling techniques: LRS and DSS. We note that all models share the same backbone and RL training. The baseline, Random, LRS, and DSS share the same reward model and only differ in the sampling techniques.
6.4 Evaluation metrics