Don’t Add, don’t Miss: Effective Content Preserving Generation from Pre-Selected Text Spans
Authors: Aviv Slobodkin
Avi Caciularu
Eran Hirsch
Ido Dagan
Venue: EMNLP 2023 Findings (2023)
Keywords: NLG
Controlled Generation
RL
Controlled Decoding
Distillation

Abstract: The recently introduced Controlled Text Reduction (CTR) task isolates the text generation step within typical summarization-style tasks. It does so by challenging models to generate coherent text conforming to pre-selected content within the input text (``highlights''). 
    This framing enables increased modularity in summarization-like tasks, allowing to couple a single CTR model with various content-selection setups and modules. 
    However, there are currently no reliable CTR models, while the performance of the existing baseline for the task is mediocre, falling short of practical utility.
    Here, we address this gap by introducing a high-quality, open-source CTR model that tackles two prior key limitations: inadequate enforcement of the content-preservation constraint, and suboptimal silver training data. 
    Addressing these, we amplify the content-preservation constraint in both training, via RL, and inference, via a controlled decoding strategy. 
    Further, we substantially improve the silver training data quality via GPT-4 distillation. 
    Overall, pairing the distilled dataset with the highlight-adherence strategies yields marked gains over the current baseline, of up to 30 ROUGE-L points, providing a reliable CTR model for downstream use.

Don't Add, don't Miss: Effective Content Preserving Generation from Pre-Selected Text Spans
Aviv Slobodkin', Avi Caciularu Eran Hirseh', Ido Dagan'
Bar-Ilan University GGoogle Research (lovodkin93, hirsch.eran)@greee. eran))@gail com avicaggoogle com daganêcs. biu.ac.
Abstract
input a text with pre-selected marked spans ("high- lights") and expects reduced and coherent version of the text, covering precisely the content of these input spans. In addition to a baseline model, the au- thors also provided crowdsourced dev and test sets, along with caammattcalll-gneenaaae silver training data, where each instance comprises of document with highlighted content and the corresponding duced text summary. The proposed adoption of CTR offers greater control over text generation, enabling modular summarization systems, where a single CTR model can be combined with var- ious content selection strategies and user prefer- ences. For example, the same CTR model could be used for generation from content selected for ei- ther generic or query-focused summarization (QFS; Dang, 2006), or for long-form question-answering (LFQA; Fan et al., 2019). It could even enable -oot---n- scenario, where customized summarics can be created, based on users' prefer- ences, as was recently demonstrated in Slobodkin et al. (2023b). Further, by excluding the subjec- tive content selection requirement of the full sum- marization task, CTR offers semantically well- defined and objective generation task, focused coherent content consolidation.
The recently introduced Controlled Text Re- duction (CTR) task isolates the text genera- tion step within typical summarization-style tasks. It does so by challenging models to gen- erate coherent text conforming pre-selected content within the input text ("highlights"). This framing enables increased modularity in summarization-like like tasks, allowing to couple a single CTR model with various content- selection setups and modules. However, there are currently reliable CTR models, while the performance of the existing baseline for the task mediocre, falling short of practical utility. Here, we address this gap by introducing high- quality, open-source CTR model that tackles prior key limitations: inadequate enforce- ment of the content- preservation constraint, and suboptimal silver training data. Addressing these, amplify the vontent-preservatiin con- straint in both training, via RI., and inference, via a controlled decoding strategy. Further, we substantially improve the silver training data quality via GPT-4 distillation. Overall, pairing the distilled dataset with the highlight- adherence strategies yields marked gains over the current baseline, up to 30 ROUGE-L points, providing reliable CTR model for downstream use.'
Being recently introduced, there currently ex- ists no high-quality CTR model, with the present baseline succeeding to cover only half of the high- lighted details while pulling much highhiiglgttted content from the surrounding context. In this paper. we aim to design an efficient CTR model of suf- ficient quality for reliable integration in modular architectures. Our proposed method, outlined in Figure 1, addresses two different shortcomings of the existing resources. First, examine methods intensify the highlights signal during training and inference, in order to yield better content preserva- tion. Second, we address the noise evident the available silver training data, improving its quality using GPT-4
1 Introduction
The abstractive text summarization task, aiming to generate accurate and coherent summaries from multiple documents, involves two principal sub-tasks: (a) identification of salient information in the input text(s) and (b) its consolidation into a coherent text. Recently, Slobodkin al. (2022) proposed an explicit decomposition of these two subtasks, particularly concentrating on the latter as an isolated task termed Controlled Text Reduction (CTR), as illustrated in Figure 2. This task takes as
"Our code is publicly available at petpps//giihhe com/ lovodking3/COR_CCR

---

Dataset
b
Fine-tuning
Inference
Input: doc with highlights
Target: silver summary
Language Model
Language Model
Cloady woutter Stantle Breameel....
Cluddy warter Staria dhvareadio...
Controlled decoding
Prompe bitbie uik. proviced wit "Aahligtr"""
Target: GPT generated summary
Highlights RL reward
Cledy warter www.errruaaneee Buanded
Figure I: Overview of our contributions, encompassing three modeling phases. Components introduced in our approach are denoted in blue. (a) We generate new target summaries using GPT-4, conditioned on the silver highlights in the original dataset. (b) During training, we fine-tune our model taking an RL approach, based on Quark (Lu et al., 2022a). (c) During inference, we employ a highlights-centric controlled decoding algorithm.
We begin our investigation by exploring strate- gies for enforcing the highlights signal. First, explore the of Reinforcement Learning (RL) during training to both accentuate the highlights signal and mitigate the impact the data noise, adapting the recemtly-ntroduueed Quark algorithm (Lu et al., 2022a), which aims to unlearn unwanted properties. Since RL methods bias the model to conform to a reward function, in addition to the training data, we hypothesize it has the potential to reduce the impact of noisy data biases the model's behavior. We then design highlight- aware decoding mechanism, following Wan al. (2023), which biases the model to better preserve the highlights content at inference time. Finally, we address the inherent noise within the available silver training data, by employing GPT-4 (Ope- nAI, 2023) to generate cleaner training data for our model, essentially performing (symbolic) distilla- tion from GPT-4.
2
Background
This section provides the needed background re- garding our task and the methods we employ.
Controlled Text Reduction Controlled Text Re- duction (CTR; Slobodkin et al., 2022) is a recently introduced task that aims to generate a reduced ver- sion of a text that exactly covers pre-determined selected content, referred to "highlights" (sec Figure 2). It effectively generalizes the sentence de- contextualization task (Choi et al., 2021), which ad- dresses only the case of rephrasing a single full sen- tence given in context to be comprehensible stan- dalone. In contrast to the full summarization task, which involves a substantial degree of subjectivity in content selection, CTR requires exact preserva- tion of modularly pre-selected content, making the task more semantically objective. At the same time, this stringent requirement for both faithfulness and coverage of the highlights, makes the task more semantically challenging than standard summariza- tion, which is obliged only to faithfulness, but not to full coverage.
Empirically, we demonstrate that each of the aforementioned strategies separately yields state- of-the-art results, surpassing the baseline model in terms of highlights content preservation. Further, we show that GPT-4 is indeed effective in gener- ating better silver training data, leading to further improvements.
Accompanying the task, the CTR authors intro- duced a manually annotated development and test sets, as well automatically-generatdd silver train set, derived from the DUC summarization dataset using summary-source alignment model (SuperPAL; Ernst et al., 2021). However, an ap- proximate 30% mismatch surfaced between the 'sil- ver highlights', namely source spans identified by SuperPAL. as related to the summary, and their re- spective summary content. Such discrepancy may cause model trained on this data to develop bi- ases toward overlooking certain types of highlights
Hence, our contribution in this paper is twofold:
1. Proposing and investigating multiple strate- gies to amplify the highlights signal in the CTR setting, addressing training, inference. and data generation.
2. Developing a high-quality CTR model, signif- icantly outperforming the available baseline.
Phttps://duc, hhttps://duc.nist.goom nist.gov/

---

Weuther experts discussed the drought Thursday and its possible causes and effecis,, saying they hoped to produce a consemsus on how much longer it will last. All of the speakers at the 1988 Drought Symposium called for more research and study.
Weather experts discussed the possible causes and effects of the drought, with the hope of producing consensus how much longer will last. John Hope, from the National Hurricane Center said the drought might end soon.
John Hope, the network's hurricane specialist and forecaster with the National Hurricane Center in Miami, suid the drought in the Southeust might be lessened or caded soon by heavier than normal hurvicane scason.
Figure 2: Demonstration the Controlled Text Reduction task. The input consists of source document and highlights (left), and the desirable oulput covers exclusively the highlighted content while preserving coherence (right). Borrowed and adapted from Slobodkin et al. (2022).
during training, as well as including salient, yet non-highlighted content. Here, we address this issue by improving the CTR training dataset.
tion (Jaques al., 2016; Ziegler et al., 2019). The objective of Quark is to teach the model generate texts of varying quality with respect the reward function. Then, at inference, the model is asked to generate high-reward outputs. The al- gorithm exhibits sutee-ofthhe-eae art results in several attribute-removal objectives, such as toxicity and unwanted sentiment, surpassing many other RL.- based text generation approaches, including PPO.
Controlling via Reinforcement Learning Re- inforcement Learning (RL) has been increasingly utilized to control various facets of text genera- tion (Pasunuru and Bansal, 2018; Yuan et al., 2019; Nakano et al., 2021), with most works relying ei- ther on REINFORCE (Williams, 1992), an algo- rithm notable for its direct yet high-variance ap- proach to policy optimization, the Proximal Pol- icy Optimization (PPO; Schulman et al., 2017), an algorithm renowned for efficiently balancing pol- icy stability and learning. There has also been a growing interest in using RL methods for reducing undesired behaviors, including toxicity (Faal et al., 2022) and redundancy (Mao et al., 2020).
In this work, we propose modifying Quark teach models unlearn biases caused by the some- what noisy training data, thereby enhancing their performance in adhering to the highlighted content.
Controlled Decoding Controlled decoding algo- rithms aim to guide models to better address crete output requirements that can be measured and enforced during decoding. To this end, various works designed constraint-sensitvee decoding meth- ods that modify the search space in accordance with the constraints (Anderson et al., 2017; Hokamp and Liu, 2017; Post and Vilar, 2018; Lu et al., 2021; Slobodkin et al., 2023a).
Following this line of work, Lu al. (2022a) recently introduced Quark an algorithm inspired by the Proximal Policy Optimization algorithm, de- signed to unlearn unwanted properties, which we leverage here. While REINFORCE and PPO are used to learn a desired policy, Quark aims to re- move or reduce specific behaviors from the learned policy, enabling it to effectively address the un- desired behaviors in the text generation process. The algorithm iteratively alternates between three steps: (1) Exploration, where the current model state generates new input-output samples from the training inputs, and then incorporates them into the stored data pool; (2) Quantization, where a prede- fined reward function is used to rank and classify the accumulated data pool into K quantiles, with a reward token assigned to each; and (3) Learning, where the algorithm maximizes the likelihood of the accumulated data pool, with the instances con- ditioned on their quantile labels. KL--divergence penalty is also applied during training to ensure proximity to the original language model distribu-
Recently, faithfulness-aware decoding mech- anism was proposed by Wan al. (2023), which involves lookahead operation during decoding. inspired by Lu et al. (2022b). At every decoding step, the algorithm projects into the future to create a complete summary commencing with the current tokens of any partially formed summary. It then se- lects tokens that provide paths exhibiting enhanced faithfulness within the search space. Formally, each token's score is calculated by:
(w: = logP(we<l + max (USH+1, (1) L(VE))
where x represents the current input tokens, is the generation step, y is the generated output, (ogs(y))) is the underlying generation score, g(-) is a faithfulness evaluation function, À stands

---

hyperparameter to regulate the emphasis placed on future predictions, and stands for the number of tokens to look into the future. Ci(Ust) is a collection of length continuations of ygt, and its size ranges from a single continuation in a greedy decoding approach, to k potential continuations in beam-search decoding. In this work, we adapt this algorithm, shifting its focus towards precise match- ing with the highlighted content rather than being (only) faithful to the entire input.
we propose alternating between two highlights- focused rewards: One that encourages coverage of highlights, for which we use ROUGE recall, and another that prioritizes adherence (faithfulness) to the highlights, for which we employ ROUGE pre- cision. Indeed, we find that this alternating reward strategy works best over the development set (see Appendix A).
3.2 Highhights--ennitie Sensitive Decoding
Data Generation with Language Models Re- cently, many works proposed using LM-generated data to directly train smaller manageable models. These efforts address various challenges, includ- ing controllability (Sclar al., 2022), model rea- soning (Zelikman et al., 2022; Hsieh et al., 2023), and language understanding (Ye al., 2022; Han et al., 2022). These works align with the Symbolic Knowledge Distillation scheme (West et al., 2022), where knowledge from the teacher model is trans- ferred via a textual dataset used for student training. Here, we employ GPT-4 to generate improved sil- ver training data for our models.
To bias the model to better preserve the high- lights content at inference time, we follow the faithfulness-aware decoding strategy from Wan et al. (2023) (see $2). Here, we adapt this method to prioritize partially formed summaries that are likely to eventually (once completed) match better the pre-selected highlights. To that end, we substi- tute the score (V 2) in Equation I with a new score, g(Vet44, Th), where Ty represents the con- catenation of the highlights. In essence, strat- egy shifts from input-focused to a highlights- focused scoring approach, while requiring both faithfulness and complete coverage. Within this framework, we evaluate two potential metrics for the computation of g(ysr (t: m/): ROUGE-L F1 and METEOR scores. We found that ROUGE-L Fi consistently matched or exceeded METEOR'S s performance, and hence adopted it for our method. Please refer to Appendix B for further details.
3 Method
This section presents the three core components of our contribution: highlights-driven reinforcement learning (RL. ($3.1), 1), highlight-attentive controlled decoding ($3.2), and the steps taken for generat- ing improved fine-tuning data with GPT-4 ($3.3), corresponding to components (b), (c), and (a) in Figure 1, respectively.
3.3 Improving Silver Training Data with GPT-4
We wish improve the quality of the CTR dataset. by leveraging the capabilities of GPT-4 (OpenAI, 2023). The existing CTR training dataset consists of summaries manually composed by experienced summarizers, while the highlights were automati- cally identified using summary-soure alignment model (see $2). As discussed, the performance of this alignment model leaves much room for im- provement.
3.1 Orrieie RL Training
To adapt Quark (see $2) for the CTR task, in- troduce highlights-focused reward, based on the ROUGE metric (Lin, 2004).3 Previous RL.-based text generation studies with ROUGE- rewards calculated ROUGE scores relative to the gold refer- ence summaries. In contrast, we propose calculat- ing the ROUGE rewards for the generated output compared the concatenated input highlights, to encourage their content preservation. Furthermore, motivate the model to balance between the task's ' two requirements, namely, covering the entire high- lights and avoiding the inclusion excessive non- highlighted content, we suggest optimizing each of these objectives separately. Inspired by the dual- reward procedure (Pasunuru and Bansal, 2018),
To capitalize on GPT-4's abilities in text genera- tion, we employ it to generate more fitting sum- maries, based on the silver highlights. To that end, we supply GPT-4 with a modular prompt, inspired by the Chain-of-Thought approach (Wei et al., 2023). This custom prompt incorporates two exemplars that deconstruct the controlled reduc- tion into three steps: (1) the listing of highlights, (2) the consolidation of highlights on a sentence- by-sentence basis, and (3) the production of the
WWo also experimented with PPO, but adopted Quark which outperformed it on the development set.

---

ultimate reduction. The detailed structure of this prompt can be found in Appendix C.
We will henceforth refer to models trained on the GPT-4-generated data as distilled, and to those trained on the original CTR data non-distilled, following the notion of Symbolic Knowledge Dis- tillation (West et al., 2022). As shown in Sec- tion $5.3, observed improved quality of the dataset, yielding better alignments between high- lights and summaries.
80
Score
ROUGE-1
60
ROUGE-2
ROUGE-L
NETEOR
2
5
examples
Figure ROUGE, METEOR, and BertScore results on 50 instances from the CTR development set of GPT-4 models, for varying numbers in-context examples in the prompt.
4 Experimental Setup
Base Model Throughout our experiments, our primary base model is the Instruction-finetune Flan-T5 large model (Flan-T33jare Chung et al.. 2022), further fine-tuned the highlights-focce focusee CTR dataset. The selection of this particular model is motivated by emergent research that indicates cntructiio-finetunee models manifest superior per- formance in tasks necessitating constrained genera- tion (Sanh et al., 2022; Wei et al., 2022; Zhou et al., 2023). We will refer to this model as Flan-T5H. where H stands for thighlight-finetuneee
utilize the Fl measure of ROUGE-L, relative to the concatenated highlights (see $2 and $3.2). We use the hyperparameters used in Wan et al. (2023). Additionally, following its tuning, we chose beam size of 8. For a more comprehensive discussion of the hyperparameter tuning, see Appendix B.
Generating New Training Data with GPT-4 As part of our methodology to improve the generation of the silver training data, we evaluated the effeet of varying the number of examples used in the few- shot prompt for GPT-4. This procedure involved the random sampling of fifty instances from the CTR development set, and their subsequent incor- poration within a few-shot prompt. The quantity of in-context examples was systematically varied between one and five, the results of which are docu- mented in Figure 3. These results show that the use of two examples delivered the most favorable re- sults across all evaluative metrics, thereby leading us to select this configuration in our experiments. We also experimented with ChatGPT and found it to be inferior to GPT-4, leading the adoption of GPT-4 this setting. For more in-depth discus- sion of these findings and additional hyperparame- ter tuning, please refer to Appendix D.
In addition to this variant of Flan-T5H, also show results of a large variant of the original pre- trained CTR baseline model, which is the Long- former Encoder-Decodee large model (LED Beltagy et al., 2020), finetuned on the CTR dataset. We will refer to this model as LEDH. Lastly, we also show the results of the few-shot GPT-4 (Ope- nAI, 2023), when guided with the same prompt used in our distillation process (see $3.3).
RL Training Our highiighs-sensitiiv RL training involves fine- tuning the anchor Flan-T5H model using Quark, combined with our highlights- drivve reward policy (see $3.1). Specifically, the first Exploration step (sce $2) utilizes the pretrained and fine-tuned Flan- T5n to generate the initial pool of samples, which triggers the Quark unlearning process, performed on Flan-T5H- Following Lu et al. (2022a), we set the number of quantiles to eight (see $2). We find, the development set, that ROUGE-L is the most effective metric for our dual-reward function, yielding the greatest improvements relative to the ROUGE-1 and ROUGE-2 metrics (henceforth, ROUGE rewards refer to ROUGE-L).
Hlighlights-Orienter
For evaluation, we adopt the evalua- tion approach outlined by Slobodkin et al. (2022), calculating several content-matching metrics be- tween the generated texts and the concatenated highlights. The rationale behind evaluating against the highlights, rather than the gold summaries, is rooted in CTR's principal requirement: an exact matching to the highlighted content, not necessar- ily to the reference summary. While the highlights and reference summary are supposed to express the
Evaluation
Highlights-Sensitive Decoding To score the de- gree of aligmment between each future completion a current generated prefix and the highlights, we

---

model
R-1
R-2
R-L
M
BertScore
Coherency
LEDn (baseline)
70.2
53.8
54.4
64.3
71.9
4.26
Non- Distillledd
Flan- 11an-T51
74.1
59.1
63.4
67.4
73.2
4.56
RL
75.6
63.1
68.9
73.6
76.8
4.34
Con. Decoding
80.7
68.6
76.1
74.2
80.0
4.40
66.4
73.4
75.1
76.8
4.42
Con. Decoding
Flan-T55
84.3
72.4
79.9
80.9
75.8
4.32
Distilled
Rl.
85.0
74.3
82.1
84.3"
81.9
4.40
Decoding
88.3*
79.1"
86.4"
84.0
84.2"
4.54
RL
86.7
76.4
84.6
84.1
81.9
4.44
Decuding
GPT-4
82.3
67.1
75.1
79.2
80.8
4.58
Table ROUGE, METEOR (M), and BertScore results on the CTR testset, compared the concatenated highlights, well as coherency results. In addition to the baseline LEDn and Flan-T5n, also evaluate the combination of Flan-T5H with the highlights-sensitie senssitvee decoding strategy ("+ Con. Decoding"), with the highlights highlights-focused RL strategy ("+ RL"", and with their combination. We also evaluate
those variants when trained with the GPT-4-generated trainset ("Distilled")) well as GPT-4 itself, in a few-shot setting with two exemplars. For each metric, the best non non-distilled and distilled Flan-T5 models are in bold, with the best overall model having an asterisk.
content, some discrepancies may occur in the gold test data. Moreover, since automatic metrics are not perfect, models that are less abstract than the numan-generated gold summaries, or that exhibit different paraphrastic abstractions, would unjustly be penalized when evaluated against the gold summarics. In Appendix E, we conduct a qual- itative analysis that further shows the advantage of directly comparing outputs with the highlights, rather than with the gold summaries.
5 Results and Analysis
5.1 None-Distilled Models
Table presents performance results, for mod- els trained over the original CTR silver training data (Slobodkin al., 2022)). Primarily, Flan-T5H exhibits superior performance compared to LEDH- This trend consistently apparent all subsequent variants, hence, we confine our reporting of subse- quent models to -lans 's variants."
In addition to ROUGE (Lin, 2004), used in Slo- bodkin et al. (2022), we also measure METEOR scores (Denkowski and Lavie, 2014), informed by recent research underscoring the correlation of this metric with human judgment concerning relevance and consistency (Fabbri al., 2021). We also re- port results on the BertScore metric, which less lexical and more semantic in nature, though not necessarily more reliable. Experimenting also with NLI-based metrics, we observed inadequate perfor- mance when applied over sub- sentenc spans (our highlights), and hence leave it for future research to develop NLI-based metrics that are robust to such settings.
We observe that further finetuning Flan-T5H via our highlights-oriented RL protocol (*+ RL" in Table 1) yields substantial improvements. Addi- tionally, augmenting Flan-T5H with the highlights- aware decoding strategy ("+ Con. Decoding" in Table 1) leads to an even larger performance im- provement across ROUGE and BertScore metrics and modest improvement in the METEOR metric. We find this sensible, given the aggressive nature of the controlled decoding approach in amplifying the highlight-signal when actually generating the output at inference time. Interestingly, the incorpo- ration of both strategies simultaneously results a slight drop in performance based on the ROUGE and BertScore metrics compared to the controlled decoding variant, yet it yields the optimal outcomes the METEOR metric. Critically, all model variants maintain human fluency score above 4.3 (out of 5), demonstrating that the fluency require- ment not compromised. This not surprising, given the impressive ability current language
Finally, we also follow Slobodkin et al. (2022)'s coherency analysis, by hiring crowd-workens to assess the coherency of 50 random samples for each model, with 5-point Likert scale (for details see Appendix F). Notably, our coherency analysis tested both coherency and fluency, following the settings the original CTR paper. This approach follows the standard common practice, where flu- ency and coherence are best evaluated manually.
"See Appendix G for the results of all the variants with LED us the backbone model.

---

models to generate fluent texts.
Faithfulness (P)
Coverage (R)
F-1
LEDD
65.8
72.9
69.2
5.2 Distilled Models
Flan-T3g
71.1
74,0
72.5
Flan-T3p (distil.)
79.1
90.8
84.6
From Table 1 we observe a noteworthy finding: when trained on the GPT-4-generated data, Flan- тhe surpasses all the non-distilled alternatives, which were trained on the original CTR data. No- tably, it also surpasses the GPT-4 model, where the latter is prompted in the few-shot setting, just as it was when generating the training data for Flan- T5H. Given the inherent limitation of GPT-4 in that it is not open-sourced and cannot be fine-tuned on our data, this puts compared to the other models, thereby obstructing its broader application.
Rl.
81.3
93.4
86.9
Con. Decoding
85.6
91.3
88,3
RL
83.4
92.3
87.7
Decoding
Table 2: Fact-wise faithfulness (P), coverage (R), and F-1 scores between generated summaries and the high- lighted spans for LEDH,, Flan-T5H. and four variants of the distilled Flan-TShr regular, with the RL finetun- ing ("+RL"), with the controlled decoding approach ("+ Con. Decoding") and with their combination. For each metric, the best models are in bold.
distinct disadvantage
the manually annotated highlights and each of the automatically-gennt gereeealyygeerrtteed variants. Our analyses veal that the GPT---generated instances demon- strate a significantly greater alignment with the annottte- instances than those from the original dataset, achieving a ROUGE-L score of 79%, as opposed to 67.9%.
We also note that Flan-T5n appears to fur- ther benefit from both the auxiliary highlights- focused RL finetuning and the incorporation of the highlights attentive decoding strategy. We find that the decoding strategy outperforms the RL ap- proach based on ROUGE and BertScore metrics, similar to the findings for the non-distilled models, while the RL approach appears to be more effec- tive on the METEOR metric. We also note that the combination of both strategies does not result in any additional advantages. Ultimately, just as in the non-distilled setting, we find that the coherency of the generated outputs remains uncompromised. In total, the combination of the highlights- focusee strategies with the improved Gen---geeerated data leads to significant improvements, outperforming the baseline model by over 30 ROUGE-L points, and resulting in a reliable and effective CTR model for future downstream use.
5.4 Manual Performance Analysis
To further evaluate the effectiveness of various com- ponents within our approach, we adopt the man- ual analysis methodology proposed in the CTR paper (Slobodkin et al., 2022). Consistent with the original authors' procedure, we select 10 ran- dom samples from the test set. Subsequently, we compute precision, recall, and F-1 scores for high- lighted information units for a selection of six mod- els utilized in this study: LEDH, Flan-T5h, the distilled Flan-T5H, and the three variants of the distilled Flan-T55 equipped only with RL training. only with the controlled decoding, and with both approaches, respectively." Our calculations cover 195 highlighted input units and approximately 180 system summary units for each model. The results of this analysis are illustrated Table 2.
5.3 Distillation Data Quality Assessment
To shed light on the distillation success, we ana- lyze the quality of the generated dataset. To that end, we first sample 10 GPT-4-generated instances and manually identify their corresponding high- lights, which we treat as "gold" highlights. Then, we apply Slobodkin et al. (2022)'s automated sil- ver annotation methodology (see $2) on the GPT- 4-generated summaries, leading to a new set of highlights for each pair of input text and GPT-4- generated summary. These highlights, combined with their corresponding inputs and summaries, rep- resent the original CTR dataset. Alternatively, the original highlights presented GPT-4, combined with the input texts and the summaries it generated, represent the new dataset. Subsequently, we pro- ceed to calculate the ROUGE-L Fl score between
We observe that Flan-T5, T5 exhibits significantly increased faithfulness to the highlights compared to LEDH, with a modest improvement in high- light coverage. Conversely, training the GPT- 4- 4-generated data markedly augments adherence the highlights well leading to significant high- light coverage, thus demonstrating the approach's inherent value.
Our findings also suggest that the application of RL finetuning primarily improves highlight cover- age. The seemingly modest contribution to faith-
Forr further details, refer to Appendix H.

---

fulness could potentially be attributed the RL mechanism's is attempt to achieve a balance between reward optimization and poherence-preservatio (due to the KL-diivegeence term), a requirement that might necessitate incorporating additional informa- tion from the surrounding context. Alternatively, the deployment of highlight-centric controlled de- coding displays more pronounced impact on ad- herence highlights. This phenomenon could potentially be attributed to the strategy's 's more rig- orous enforcement of highlight constraints, thereby enforcing limited deviation from the highlights. The modest advancement in coverage could be at- tributed to the lookahead mechanism of controlled decoding. At every generation step, the algorithm favors tokens whose 'greedy' completion leads to better highlight adherence, rather than exploring multiple potential trajectories for each candidate akin to k-beam style. In doing so, this method inadvertently neglects additional, more suited can- didates, whom a beam search could capture and thereby enhance coverage. Lastly, the combina- tion of both strategies results in mid-point per- formance between the individual gains attained by each strategy in both faithfulness and cover- age. These findings indicate that while merging the strategies does enhance the weaker model's perfor- mance in each metric, it also has a trade-off effect on the other metric. Future research will explore methods to harness the complementary aspects of these strategies while mitigating this trade-off ef- fect.
the decoding strategy may be integrated into the RL training's sampling phase, possibly increasing the model's awareness of the decoding constraints during training. Our performance analysis, detailed in $5.4, further supports this hypothesis by demon- strating that each technique amplifies different as- pects of the highlights: one notably improves their coverage while the other augments adherence, sug- gesting the exploration of this potential synergy in future research.
Furthermore, the existing implementation of highlight-centric controlled decoding is computa- tionally demanding, given its requirement to gen- erate a complete completion for every candidate token at each generation step. Wan al. (2023) motivated the need to generate entire summaries by the expectation of current faithfulness metrics for full summary inputs. Yet, as our method does not employ faithfulness metrics, would be insight- ful to explore how to leverage partial summaries, rather than complete ones, and thus to significantly reduce the computational overhead.
Finally, given the high-quality highlight adher- ence and coverage exhibited by our best models, investigating their incorporation into modular sum- marization pipelines emerges as a promising re- search direction. We suggest exploring this direc- tion in future research.
7 Conclusion
In this study, we addressed the lack of high- quality Controlled Text Reduction (CTR) model by focusing on two pivotal aspects: the amplification of the highlight signals and the mitigation of noise within the training data. We started by proposing two distinet strategies aiming at augmenting the highlights signal. The first strategy emphasized this signal during training, where we combined an RL approach with a custom highlights-oriented re- ward. The second strategy was introduced during inference, where employed controlled decod- ing mechanism that prioritizes generation paths ensuring higher adherence to the highlights. Fur- thermore, we addressed the intrinsic noise in the CTR dataset by generating new instances using GPT-4, significantly enhancing the dataset quality. Empirical evidence shows the effectiveness of our proposed methodology. Each of our highlight- centric strategies individually led to significant improvements over the baseline model in terms of highlight-matching capabilities. Additionally,
6 Discussion and Future Work
Our empirical assessments reveal the merits of each of the proposed methods. Specifically, the utility of the distillation process emerges as complementary to both reinforcement learning (RL) training and controlled decoding. The latter methods still have room to play in both enforcing the content preser- vation constraints beyond the sheer training data, well as in overcoming a certain level of noise that persists also in the GPT-4 generated data (as shown in $5.3).
Conversely, the combination of controlled decod- ing and highlight-centric RL approach did not yield improvements over the berferipe method (controlled decoding, in our experiments). Given these strategies impact different stages of the gener- ation process, namely training and inference, they may possess the potential for synergy. For example,

---

training on the GPT-4-generated data yielded fur- ther improvements, outperforming each of the non-distilled variants, trained on the original CTR dataset. In total, our gighest-performing models achieved state-of-the--at results, outperforming the baseline by more than 30 ROUGE-L points, while also preserving comparable levels of coherency. Fu- ture work would focus improving the combina- tion and efficiency of the different components, as well as on the incorporation of our pest-pefoonnig models in modular summarization pipelines.
saults, burnings, and beheadings, are beyond any justification.
In addition to the loss we suffered as a nation and as human beings due to this violence, many of us feel abandoned and betrayed by members of our research community who did not reach out and were even reluctant to publicly acknowledge the inhumanity and total immorality of these acts.
We fervently call for the immediate release of all those who have been taken hostage and urge the academic community to unite in condemnation of these unspeakable atrocities committed by Hamas, who claim to be acting in the name of the Pales- tinian people. We call all to join us in advocating for the prompt and safe return of the abductees, as stand together in the pursuit of justice and peace.
8 Limitations
Although the risk involved in our work is minimal, like other advanced language generation models available today, we cannot ensure that our model will consistently produce accurate information, de- spite our attempts to utilize clean highlighted data and controlled decoding. Hence, it is crucial to exercise caution when employing the model in real- world scenarios and thoroughly test it before de- ploying it.
Acknowledgements
This work was supported by the Israel Science Foundation (grant no. 2827/21), and a grant from the Israel Ministry of Science and Technology.
In addition, the controlled decoding method we use is limited by its slow speed, which may hinder its practicality for production systems. Addition- ally, the rigid constraints imposed during decoding restrict the model's linguistic flexibility and creative potential, potentially limiting its suitability for generating varied or innovative outputs. These considerations highlight the need to carefully eval- uate the trade-offss before implementing controlled decoding in production environments, and hence should be further studied in future research.
References
Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. 2017. Guided open vocabulary im- age captioning with constrained beam search. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 936- 945, Copenhagen, Denmark. Association for Compu- tational Linguistics.
Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long-document transformer.
9
Ethics Statement
Eunsol Choi, Jennimaria Palomaki, Matthew Lamm, Tom Kwiatkowski, Dipanjan Das, and Michael Collins. 2021. Decontextualization: Making sen- tences stand-alone. Transactions of the Association for Computational Linguistics, 9:447-461.
This paper is submitted in the wake of tragic ter- rorist attack perpetrated by Hamas, which has left our nation profoundly devastated. On October 7, 2023, thousands of Palestinian terrorists infiltrated the Israeli border, launching a brutal assault 22 Israeli villages. They methodically moved from home to home brutally torturing and murdering more than thousand innocent lives, spanning from infants to the elderly. In addition to this horrifying loss of life, hundreds of civilians were abducted and taken to Gaza. The families of these abductees have been left in agonizing uncertainty, no infor- mation, not even the status of their loved ones, has been disclosed by Hamas.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Web- son, Shixiang Shane Gu, Zhuyun Dai, Mirac Suz- gun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang. Gaurav Mishra, Adams Yu. Vincent Zhao. Yanping Huang. Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts. Denny Zhou, Quoc V. Le, and Jason Wei. 2022. Scaling instruction-finetunes language mod- els.
Hoa Trang Dang. 2006. DUC 2005: Evaluation of question-focused summarization systems. In Pro- ceedings of the Workshop on Task-Focused Sunma- rization and Question Answering, pages 48-55, Syd-
The heinous acts committed during this attack, which include acts such as shootings, sexual as-

---

ney, Australia, Association for Computational Lin- guistics.
Ximing Lu, Sean Welleck, Jack Hessel, Liwei Jiang. Lianhui Qin, Peter West, Prithviraj Ammanabrolu. and Yejin Choi. 2022a. QUARK: Controllable text generation with reinforeed unlearning. In Advances in Neural Information Processing Systems (NeurIPS).
Michael Denkowski and Alon Lavie. 2014. Metcor universal: Language specific translation evaluation for any target language. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 376-380, Baltimore, Maryland, USA. Association for Computational Linguistics.
Ximing Lu, Sean Welleck, Peter West, Liwei Jiang, Jungo Kasai, Daniel Khashabi, Ronan Le Bras, Lian- hui Qin, Youngjae Yu, Rowan 7ellers, Noah A. Smith, and Yejin Choi. 2022b. NeuroLogic a aeesque de- coding: Constrained text generation with lookahead heuristics. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 780-799, Seattle, United States. Association for Computational Linguistics.
Ori Ernst, Ori Shapira, Ramakanth Pasumuru, Michael Lepioshkin, Jacob Goldberger, Mohit Bansal, and Ido Dagan. 2021. Sunmary-source proposition-level alignment: Task, datasets and supervised baseline. In Proceedings of the 25th Conference on Computa- tional Natural Language Learning, pages 310 -22, Online, Association for Computational Linguistics.
Farshid Faal, Ketra Schmitt, and Jia Yuan Yu. 2022. Re- ward modeling for mitigating toxicity in transformer- based language models. Applied Intelligence.
Ximing Lu, Peter West, Rowan Zellers, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Neuro- Logic decoding: (un)supervised neural text genera- tion with predicate logic constraints. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 4288-4299 Online, Association for Computational Linguistics.
Alexander R. Fabbri, Wojeiech Kryseiński, Bryan Mc- Cann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021. SummEval: Re-evaluating Summariza- tion Evaluation. Transactions of the Association for Computational Linguistics, 9:391-409.
Yuning Mao, Yanru Qu, Yiqing Xie, Xiang Ren, and Jiawei Han. 2020. with maximal marginal relevance- guided reinforce- ment learning. Proceedings of the 2020 Confer- ence on Empirical Methods in Natural Language Processing (EMNLP), pages 1737-1751.
Angela Fan, Yacine Jernite, Ethan Perez, David Grang- ier, Jason Weston, and Michael Auli. 2019. EL15:
summarization
Long form question answering. In Proceedings of the 57th Annual Meeting of the Association for Com- putational Linguistics, pages 3558-3567, Florence, Italy, Association for Computational L Linguistics.
Retichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou. Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. 2021. Webgpt: Browser- assisted question-anseee answering with human feedback.
Jesse Michael Han, Igor Babuschkin, Harrison Edwards, Arvind Neelakantan, Tao Xu, Stanislas Polu, Alex Ray, Pranav Shyam, Aditya Ramesh, Alec Radford, and Ilya Sutskever. 2022. Unsupervised neural ma- chine translation with generative language models only.
Chris Hokamp and Qun Liu. 2017. Lexically con- strained decoding for sequence generation using grid beam search. In Proceedings of the 55th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 1535-1546, Vancouver, Canada. Association for Computational Linguistics.
Ani Nenkova and Rebecca Passonneau, 2004, Evaluat- ing content selection in summarization: The pyramid method. In Proceedings of the Human Language Technology Conference of the North American Chap- ter of the Association for Computational Linguistics: HLT-NAACL 2004, pages 145-152, Boston, Mas- sachusetts, USA. Association for Computational Lin- guistics.
Cheng-Yo Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alexander Ratner, Ranjay Krishna, Chen-Yu Yu Lee, and Tomas Pfister. 2023. Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes.
OpenAL 2023. Gpt-4 technical report.
Ramakanth Pasunuru and Mohit Bansal. 2018 Multi- reward reinforced summarization with saliency and entailment. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 2 (Short Papers), pages 646-653, New Orleans, Louisiana. Association for Computa- tional Linguistics.
Natasha Jaques, Shixiang Gu, Dzmitry Bahdanau, José Miguel Hernández-Lobato, Richard E. Turner, and Douglas Eck. 2016. Sequence tutor: Conserva- tive fine-tuning of sequence generation models with kl-control.
Chin-Yew Lin. 2004. ROUGE: A package for auto- matic evaluation of summaries. In Text Summariza- fion Brunches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Matt Post and David Vilar. 2018 Fast lexically con- strained decoding with dynamic beam allocation for neural machine translation. In Proceedings of the 2018 Conference of the North American Chapter of

---

the Association for Computational Linguistics: Hu- man Language Technologies, Volume (Long Pa- pers), pages 1314-1324, New Orleans, Louisiana. Association for Computational Linguisties.
Peter West, Chandra Bhagavatula, Jack Hessel, Jena Hwang, Liwei Jiang, Ronan Le Bras, Ximing Lu, Sean Welleck, and Yejin Choi. 2022. Symbolic knowledge distillation: from general language mod- els to commonsense models. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, pages 6602-4625, Seat- tle. United States. Association for Computational Linguistics.
Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, 7aid Alyafeai, Antoine Chaffin, Arnaud Stiegler. Teven Le Scao, Arun Raja, Manan Dey, M Saiful Bari. Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szezechla, Taewoon Kim. Gunjan Chhablani. Nihal Nayak. De- bajyoti Datta, Jonathan Chang. Mike Tian-Jian Jiang, Han Wang. Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan. Tali Bers, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M. Rush. 2022. Multi- task prompted training enables zero-shot task gener- alization.
Ronald J Williams. 1992. Simple statistical gradient- following algorithms for connectionist reinforcement learning. Machine learning. 8(3):229-256.
Jiacheng Ye. Jiahui Gao, Qintong Li. Hang Xu. Jiangtao Feng, Zhiyong Wu, Tao Yu, and Lingpeng Kong. 2022. ZeroGen: Efficient zero-shot learning via dataset generation. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, pages 1663-11669, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proxi- mal policy optimization algorithuss. arXiv preprint arXiv:1707.06347
Xingdi Yuan, Mare-Alexandre Côté, Jie Fu, Zhouhan Lin, Chris Pal, Yoshua Bengio, and Adam Trischler, 2019. Interactive language learning by question an- swering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Lunguage Processing and the 9th International Joint Conference on Natu- ral Language Processing 2796-2813, Hong Kong, China. Association for Com- putational Linguistics.
Melanie Sclar, Peter West, Sachin Kumar, Yulia Tsvetkov, and Yejin Choi. 2022. Referee: Reference- free sentence summarization with sharper controlla- bility through symbolic knowledge distillation.
Aviv Slobodkin, Omer Goldman, Avi Caciularu, Ido Dagan, and Shauli Ravfogel. 2023a. The curious case of hallucinatory unanswerablity: Finding truths the hidden states of over-confident large language models.
pages
Eric 7elikman, Yuhuai Wu, Jesse Mu, and Noah Good- man. 2022. STar: Bootstrapping reasoning with rea- soning. In Advances in Neural Information Process- ing Systems.
Aviv Slobodkin, Niv Nachum, Shmuel Amar, Ori Shapira, and Ido Dagan. 2023b. Summhelper: Col- laborative human-computer summarization.
Aviv Slobodkin, Paul Roit, Eran Hirsch, Ori Ernst, and
Wangchunshu Zhou, Yuchen Eleanor Jiang, Ethan Wilcox, Ryan Cotterell, and Mrinmaya Sachan, 2023. Controlled text generation with natural language in- structions.
Ido Dagan. 2022. Controlled text reduction. Pro- ceedings of the 2022 Conference on Empirical Meth- ods in Natural Language Processing, pages 5699 5715, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Chris- tiano, and Geoffrey Irving. 2019. Fine-tuning lan- guage models from human preferences. arXiv preprint arxiv:1909.08593.
David Wan, Mengwen Liu, Kathleen McKeown, Markus Dreyer, and Mohit Bansal. 2023. Faithfulness-awee aware decoding strategies for ab- stractive summarization. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2864-2880, Dubrovník, Croatia. Association for Computational Linguistics.
A Reward Tuning
We train the modified dual-reward Quark model with three pairs of highlights oriented ROUGE re- wards: ROUGE Precision+ROUGE F1, ROUGE ROUGG FI, and ROUGE Precision ROUGE Recall. From Table 3, which shows the ROUGE scores on the CTR development set, it arises that alternating between ROUGE precision and ROUGE recall rewards yields the best results.
Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, An- drew M. Dai, and Quoe V. Lc. 2022. Finetuned language models are zero-shot learners.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. -hain-of-thought prompting elic- its reasoning in large language models.

---

R-L Rewards
R-1
R-2
R-L
M
BertScore
ROUGE-1
Precision + Fl
74.0
60.7
67.3
69.8
76.3
Recall
74.1
60.9
67.6
70.4
76.2
Only Fl
74.4
61.2
67.3
70.0
76.4
Metric Score
Precision + Recall
76.7
64.0
70.4
74.0
77.7
regular decoding
78
controlled decoding
Table ROUGE, METEOR (M), and BertScore results of the Flan-T5h model, combined with the RI strategy, on the CTR development set. We experimented with dif- ferent pairs of alternatin highlights-oricntedd ROUGE-L rewards, as well only single ROUGE-L Fl reward.
76
2
Beam size
ROUGE-2
R-1
R-2
R-L
M
BertScore
68
METEOR
77.0
60.3
61.7
71.3
74.3
Metric Score
66
ROUGE-L
78.3
63,9
72.3
71.3
76.4
regular decoding
64
controlled decoding
Table 4: F1 results for ROUGE, METEOR (M), and BertScore compared the concatenation of the high- lights on the dev set. The results are for Flan-T5H with a beam size of 2, incorporating two variants of the highlights-centric score during decoding: METEOR (FI) compared to the highlights, and ROUGE-L (FI) compared to the highlights. Bold marks highest results on each metric.
Beam size
ROUGE-L
77.5
75.0
Metric Score
72.5
regular decoding
70.0
controlled decoding
B Highlights-Focus Controlled Decoding Hyperparameter Tuning
65.0
For our highlights-oriented decoding strategy, we experiment with two scores: METEOR score and ROUGE-L F1 score. These are juxtaposed against the concatenated highlights to assess the potential completion's 's adherence to the highlights. Table 4 shows the results the CTR development set. We note that the ROUGE-L score consistently offers superior or least equivalent performance in com- parison to the METEOR score. Therefore, based on these findings, we make the informed decision to adopt the ROUGE-L score as our primary evalu- ation metric in all subsequent experiments. Further- more, we undertake series of experiments with varied beam sizes encompassing 2, 4. 6, and 8. The outcomes, observed on the development set, illustrated in Figure Corroborating the findings of Wan al. (2023), we discern a marginal impact of the increase in beam size on regular decoding. However, its influence on the highlight-focused controlled decoding escalates in parallel with the beam size across all metrics. In light of these find- ings, opt for beam size of 8 for the remainder of our experiments.
2
4
Beam size
METEOR
Metric Score
regular decoding
controlled decoding
Beam size
Bertscore
80
Metric Score
78
regular decoding controlled decoding
76
74
2
Beam size
Figure 4: The scores of fine-tuned Flan-T5H with trolled decoding compared to regular decoding at vari- ous beam sizes. present comparison of the gener- ated summary the highlights concatenation.
C GPT-4 Prompt
For deploying GPT-4, we use a prompt that con- sists of some basic instructions, as well as two
exemplars. The in-context examples consist of a modular generation, where we separate the task

---

Model
R-2
R-1.
BerSSooe
R-1
R-2
R-L
M
BertScore
GPT-3.5
regular
59.9
40.7
47.7
49.7
70.0
w'o lista
84.3
68.4
76.4
81.4
81.8
modular
72.7
61.1
68.1
75.0
75.7
with list.
84.8
71.3
77.1
83.6
81.8
GPT4
regular
71.0
55.1
64.5
63.1
76.4
modular
81.0
66.0
73.0
76.3
80.5
Table 6: ROUGE, METEOR and BertScore results, compared to the concatenated highlights, on 50 in- stances from the CTR development set of GPT-4 models, once when the prompt does not contain highlights enu- meration the current instance (" ("wto list, ") and once when it does contain it ("with list;") ").
Table 5: ROUGE, METEOR (M), and BertScore re- sults 50 instances from the CTR development set of GPT-3.5 and GPT-4 models. For each model, we prefixed an exemplar to the prompt, in two fashions: "regular". where the exemplar simply generated the cus- tom summary; and "modular", where the exemplar first performed the intermediate sentence-by cenennce-b--seteence be-eeneeceee high- lights consolidation, before generating the final sum- mary.
In the first example, the non-distilled model achieved ROUGE-L score of 58.6 when com- pared to the gold summary, while distilled coun- terpart received a score 46.5, Despite the appar- ent superiority of the non-distilled model, we note that while it did cover all the highlighted spans, it also added several non non- aon-higlighted segments ("an Atlanta-bass baaed cable network", "the next business of the scientist", "the network's" and "look for" in the context of looking for cause and effect). In contrast, the distilled model, besides capturing all highlighted content, only added "look for". Con- sequently, the distilled model delivered a more fit- ting output, contradicting its supposed deficiency in terms of the ROUGE-L metric relative to the gold summary. Alternatively, when compared to the concatenated highlights, which yielded scores of 68.8 for the non-distilled model, and 78.9 for the distilled model, the results are more reflective of the actual performance.
into three steps: (1) highlights extraction and enu- meration, (2) highlights consolidation sentence-by- sentence, and (3) generation of the final reduction (see Figure 6). Additionally, we already perform the highlights extraction step for the current exam- ple, leaving only the consolidation and generation of the final reduction to the model.
D Prompt-Tuning
To improve the quality of the CTR trainset, we test both GPT-4 and GPT-3.5 (ChatGPT), both with modular prompting and with regular prompting, where each exemplar's answer consists solely of the final reduction. From Table 5, which shows the ROUGE and METEOR scores on 50 instances of the CTR development set, we observe that indeed the modular approach substantially improves both models' performances and that GPT-4 is superior to ChatGPT in handling the task. Additionally, from manual analysis of GPT-4's generations, we find that occasionally it misses highlighted span in the hightights-listing step. leading to its absence from the final summary. Consequently, we also experiment with a prompt that already incorporates the highlights-listingg step, leaving only their con- solidation and the generation of the final reduction to the model. This version indeed which improves the model's 's performance (see Table 6), making it the final version use.
In the second example, the distilled model amounts to 83.3 comparison to the concatenated highlights, as opposed to a mere 43.2 when com- pared to the gold summary. Interestingly, despite the fact that its output nearly covers all the high- lighted spans (except for "with a chance of showers Finla."i and "in Helsinki" in the context of where the eclipse would end), and introduces very minimal -on-highlighted information ("thousands of", "3,000" and "the national airline"), it still re- sults a lower ROUGE-L score when aligned with the gold summary. This discrepancy can be attributed to two significant factors. Firstly, the distinction in abstraction techniques employed be- tween the gold summary and the generated output can contribute to the reduced scores. Secondly, the gold summary introduces information that is absent from the original input text, and therefore from the highlights ("on July 20"). Both these elements can explain the divergence between the model's 's performance as per the ROUGE-L metric
E Qualitative Analysis
Figure 7 exhibits two instances from the test set, consisting of input texts along with their corre- sponding highlights. Accompanying each instance are the outputs generated by the non-distilled Flan- TSH, the output generated by the distilled Flan (5nn,T5,, and the original gold summaries.

---

and its actual competency. This discrepancy can be resolved when directly comparing the generated summary to the highlights, which offers more accurate reflection of its quality.
lighted unit conveying the same information, de- termine whether the summary unit is mentioned in the highlights (TP) or not (FP), and then calculate the (micro-)precisiion to the highlighted content. For the recall calculations, we also count the number of False Negative (FN) summary facts, compared to the facts in the highlights, namely highlighted information units that were absent from the system summaries. Then, combined with the TP count, we calculate the (adicro-)recall of the highlighted content.
F Fluency Human Annotation Protocol
We ask growd-workend to rate the fluency of the texts generated by the baseline supervised model, the PPO model and the dual-reward Quark model. Our group of crowd-workers consists of reliable workers that have shown a good understanding of different semantic tasks including summarization in previous experiments. To evaluate, we randomly select 100 documents from our test set and evalu- ate their corresponding generated text by the three aforementioned models (300 samples in total). We design simple Amazon Mechanical Turk inter- face, where we present each time one of the 300 samples (see Figure 5). Following Slobodkin et al. (2022) we use a 5-point Likert scale to evaluate the fluency of the generated summaries. Additionally, we add criteria explaining each score, to reduce ambiguity and ensure consistent ratings (see Fig- ure 5). Assessing average response period of 30 seconds, we priced each response with 10
G Results of All Variants with LED, as the Backbone Model
Table 7 shows performance results for all the vari- ants explored in this work, with LED# as the back- bone model.
H Performance Analysis Settings
To further examine the efficiency of our different components, we follow Slobodkin et al. (2022) and manually assess several of our models two lev-
els: (1) precision to the highlighted content and (2) recall of the highlighted spans. For that, compare each system summary span the source highlights. To that end, we randomly select 10 samples from our test set, with their corresponding system summaries (one for each of the models 50 in total). Then, following the notion of Sum- mary Content Unit (SCU) in the Pyramid method for summarization evaluation (Nenkova and Pas- sonncau, 2004), extract such units from both the summary and the source highlighted spans ing the Summary Evaluation Environment (SEE) interface, described in their paper.
Then, to calculate the precision, for each sum- mary unit, we manually search for matched high-

---

Instructions
this back you are ankid rato the quality ot passage
correcty
precco following
Hoad the peccagn.
(weret) by luency.
De fniiiiiionn of Pluency
This noting
quallty
the will witton and grommnc coneed, they togehor
Corsiliee Me,
The scale shoud bo:
The summ summary connal only small parts taat are conerom: erom with many grammatcaal einons and sentancos that togother soorty The summary ia somewhaal conaront thouh e emee containe paveral gra grammatica omors or sortances that tonttfft woll.. grammetical
durmttary mostlyy conoront
oounds natural onough.
Select an aptian
Major variety communicat zate tis iock fondness coraon colleegues may do him good. With luck & recioned Majo's image as lescen songuee wimp may undorgoo corrctioo Major is genner escedialy ofter thts worlff revoll the sooden---ppo the Chrfonchnch unch beclecion whore Consorvaiie majonty of23 010 years gonora siection was conwenod nod into 16,427 Liboral Zomocras majority privatc they reax Thoir syntax dispppears. they sneer shout.
Majon's comersstion
Brunson, TN potiicall ochot
eightlottewerds
1.ont
fluntt
fucnt
Figure 5: Example data collection interface used by the croke-workers evaluate the fluency of summaries.
model
R-1
R-2
R-L
M
Bertscore
LEDH
70.2
53.8
54.4
64.3
71.9
Non Distilled
RI.
70.8
56.1
56.5
70.9
73.7
Decoding
77.4
64.7
71.9
69.5
77.8
RL
75.4
60.9
67.5
68.6
76.7
+ Con. Decoding
LEDI
77.9
64.0
69.7
75.6
74.0
Distilled
RL
76.6
61.7
66.8
75.4
78.2*
Con. Decoding
81.4"
68.5*
77.9*
75.5*
77.1
RL
78.1
63.8
73.0
73.1
75.6
+ Con. Decoding
Table ROUGE, METEOR, and Bertscore results on the CTR testset, compared to the concatenated highlights, of all the different variants tested in this work, with LED,; the backbone model. In addition the baseline LED,, -eenssiiv-eeeevve decoding strategy ("+ Con. Decoding"), RL strategy ("+ RL"), and with their combination. We also evaluate all those variants when trained with the GPT-4-distilled trainset (" ("Distilled"). For each metric, The best non-distilled and distilled LED# models are in bold, with the best overall model having an asterisk.
we also evaluate the combination of LED, with the with the highlights-focure

---

In this task, you presented with passage, where some parts "highlighted" (nanely, there are <highlight_ <highlight_end" tokens before and af after each such span).
Your job generate summary that covers and only the "highlighted spans.
2
Example1:
Passage: shighliigt__tarttben Johnson spent his homecomingghihhhiight..en. in seclusion, thighlight_startwee thechighlight_endd Olympic shighlightt_ttrrbbglld medal mndchighlight_eme the <highligtt_starthher welccoeehhghllightt.em as Canadians bemoaned the fate of the sprinter who failed the drug test.
5
6
7 Answer: The highlighted spans are:
8
1. Ben Johnson spent his homecoming
9
10 The highlights spans are combined as follows:
11 Spans 1,2,3,4 are combined to form sentence 1: Ben Johnson spent his homecoming without the gold medal and hero's 's welcome.
12
13
So, the answer is:
14
Ben Johnson spent his homecoming whoht the gold medal and hero s welcome.
15
16
17 Example2:
18
Passage: Japanese writer thighLight_startXKe Ishiguro thechighlight_endds 1989 itghhiightttttrtBokee Prizechighlight.ene Britain's top literary award, ttighlight_startofo his novel "The Remains of the Dayshighlight_endby, judges htw.highhhhsstrrammuneee Thursdaychigllight.emm
19
20
21 Answer: The highlighted
22
1. Kazuo Ishiguro the
23
24
The highlights spans combined follows: It announced Thursday that Kazuo
Spans 1,2,3,4 combined to form Ishiguro won the Booker Prize for his novel "The Remains of The Day".
So, answer is:
28 It Remains of The Day".
Thursday that Kazuo Ishiguro won the Booker Prize for his novel "The
31 Now your turn:
32 Passage: <highlight_start>Thee motion picture industry's most coveted awardshighlight_ends
htariiigh_staatttscrriiiii created 60 years ago 1,816 the statuettes have been produced so far, far.chighlighttend
shighlight_start>waa
34 Answer: The highlighted spans are:
35
1. The motion picture industry's s most coveted award
36
37
The highlights spans are combined as follows:
EExmmle prompt provided GPT-4. The prompt consists basic instructions, in-context examples, and the instance input. The examples demonstrate modular pipeline, where we first extract the highlights, then consolidate them centence-by-senteee and lastly generate the final reduction.
Figure 6:

---

Input: Weather experts discusssed drought Thursday and possible causes and effects, said they hoped produce consensus
much longer will last. All of the speakers at the 1988 Drought Symposium called for more research and study, "There's an enomous number of passibilizies that one could chase. The of... extend range analysis and look cause and effeet'' said David Rodenhuis, director of the federal Climate Analysis Center Washingten. The symposium attended by 13 state and national experts was sponsured by The Weather Channel, an former forecaster with the National Humricane Center in Miami, Fla., said the drought in the Southeast might be lessened or enied in the next few monts by heavier than normal hurricane season. Ioope said current weather conditions point more tropical storss, more
next business the scientist
eliminatc the
Atlanta bused cuble network, which bruadcast meeting live. John Hope, the netwerk's hurricane specialist
hurricand offect drought is variable, depending on the intoneity and speed of the storm. Neverthekess, he said, some Southeastern states get much as 25 pecent of their rainfall ini soms momihs hurricanes, Geongi state climatologist Gayther Plummat told symposium he thought the position of planet Bupiter something with drought in the Southeast However, Redenhuis stid he was skeptical about Plummer's thesis. "It's very accessury to be caatioss and analytical about the issuc," he said.
should become hurricaccs,
hurricane scsson. He stressed
The 1988 Drought Symposium was sponsored by The Weather Channel, an Atlento-based cable network. 13 state and national experts ottended. David Rodenhuis, director of the faderal Climatc Analysis Center in Wishingtes, said the business of the scientist to climinate possibilitiis and extend range of analysis and look for cause und effect. John Ilppe, the network's is hurricane specialist, said that the drought in the Southeast might be lessened or ended by heavier than nurmal hurricane senson. Some Southeastem states get as much as 25% of their rainfall in some menths from hurricunes. Georgia state climatologist Gayyhhrr Plummer said he thought position of Jupiter had something do with the drought, Rodenhuis was skeptical.
Not distilled:
The 1988 Drought Symposium was held to eliminate the possibilitie of the drought and extend the range of amalysis to lock for cause and effect. according David Rodenhuis, director of federal Climate Amilysis Center Washington. The symposium, attended by state national experts, was sponsored by The Weather Channel, John Hope, hurricane specialist, said that the drought in the Southest might be lessened or ended by heavier than normal hurricane season. Some Southeastem states get as much as 25% of their rainfall in some months from hurricanes, Georgin state climatologist Gnyther Plummer, who spoke at the symposium, thought the position the planet Jupiter had something do with the drought, but Rodenchides was skeptical about his thosis.
Distilled: :
Gold: Today's 1988 Drought Sympocium, spansored by The Weather Channel, was attended by 13 state and national Weather experts. David Rodenbuis, director of federal Climate Analysis Center Washington, stressed the need eliminate couse and-efiect possibilities and extend the range analysis. Geongia state
elimatollogitt Gayther Plammer thought the drought in the SE related the position of Jupiter, but Rodenbuis remained skeptical. Hurricane specialist John Hope suggested the SB drought might be lossened or ended by a heavier than normal hurrkanc scason. Some SE states get as much as 25% of their rainfall some months from hurricunes.
Cloony weather Saturday Birestened to our the show for thousands of Finnishi and Foreign Rkygarers hoping glimpso a total solat celipec in this the midniggt sun. The weather forecast took a turn for the worse in the evening, when the Finnish Metcorological Services predicted cloudy weather (with climno of Showers for castem Finland on Sunday, In the castern town of Jocnsuu, iclevision news breadesst latc Saturday showed was slready cloudy there with a light drizzle falling. The solar cclipse in Finland starts at 4:03 a.m. Sunday (9:03 p.m. EDT Sarunday). At that time, the moon will begin graduilyy moving between the the sun. total colipse begins uncovering more and mare of the sun. The cclipse aands at 5:45 a.m. in Helsinki About 10,000 peoplé including 3,000 fereigners haae converged on Joansu shout miles from Soviet bordor. There conditions there are considered especially good for viewing the celipse weather permitingg In Helsinki, eclipse phase will occur 16 minutes after sunrise, when the sun is only degree above the horizon. At Joensu. 310 miles northeast of Illlsnkii, the sun will be 5 degrees ahove borizon in total pluse at better angle for watchers. The sun rises unusually early during summer in extreme northern latitudes where
Input:
m. Helsinki and last 83 seconds, After the total plase
aclipse, the moon move away.
total
Finland located. The sun comes uncuually late during winter. Ten months ago, Joensu hired an "ecligne secretary" to handle arrangements for the expected influx visitors, that official, Marjut Cadia, said she had underestimated made for this event, and our esar stock is finished too.' she said in a television interview Suturday. During the past week, newspapers, television and radio have been. information about solar eclipses well as udvice spectators not stare direcily the sun. Some eclipse viewers won't have to wurry about the clouds, because they will be above them. Finnair, the national airline. has urranged dozen special nights for eclipse waschers, and private companies with small planes will provide more. The eclipse will he sotal in are about miles wide from the northern Baltic Sea and southeust Finland, across Kola Peninsuls northeast Soviet Union, to the Alcutian Islands near Alasko, Scientists will conduct several experiments during the colipse, including gravity test measurements of the radius the However, scientific interest this cclipse
interest the event. "We completely sold out the 10,000 special eyeg lasses
than lenger eclipses, such as the soven-minute cclipso seen from northern Kenya on June 30, 1973, Seppo Limnalaoto of the Unsa Astronemical Asseciation said the best place for obser-ations of this celipse will be the northeast Soviet Union. The Jast tosal sclipse to seen Finland uccurred in 1945.
Not distilled; About 10,000 people, including 3,000 foreigners, have canvergod on Joensuu, 50 mills from the Soviet booder, to view total solar celipse. Conditions there ure considered especially good for viewing. The total eclipse will begin at 4:52 a.m and last 83 seconds, The last tocal eclipse to be seen in F'inaand occurted 1945, eclipse total in an are about 125 miles wide. Thu clouds will above viewers, Finnair, national airline, has arranedd special nlihts for celipee watchers.
Distilled: : Cloudy wenther threatened to mar show for thousands Finnish and foreign skygueets hoping glimpse total cclipse in Finland. The total
oclipse bagins at 4:52 a.m. in Helsinki and will last 83 seconds. The celipse ends at 5:45 a.m. In Joensu, about 10,000 people, including 3,000 foreigners, have converged on the town, with conditions considered especially good for viewing the celipse. Some edlipse viewers won't have to wory about the clouds, as they will be above them. Finnair, the national nirtine, aranged special flights for eclipse watchers, with the cclipse heing an are about 125 miles wide, last oclipse to be in Finland oocurred 1945.
Gold: Cloudy, miny weather threatened to ruin viewing of solar eclipse in Finland on July 20. Best viewing. weather permitting would be the eastern town of Joensuu and about 10,000 people, including foreigners have guthered there. The eclipse begins at 4:52 in lelsinka and ends 5:45 am with the total phase lasting for 83 scconds. Finnair has arranged llighs for some viewers to go above the weather to view the cdlipse, The celipse will be total in an a 125 mile wide are in northern aretic aress. The last lotal colipse viewed Finland was in 1945.
Figure 7: Examples including the input, gold output and predictions with various baselines.


Reviews:
[{"review_id": "DHRbcRitfY", "reviewer": "Reviewer_AKMy", "summary": "", "questions": "Is there any way to evaluate the quality of original training data and the GPT-4 generated training data?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for appreciating our work.\n\n***Components borrowed for prior work***\n\nIn this study, we sought to develop a high-performing CTR model, given the appeal of the task for downstream use vs. the mediocre performance of the available baseline model. Our methodology was to comprehensively investigate appealing paradigms for biasing the generation model to comply with the relatively strict constraints of the CTR task. Developing a suitable RL method for the CTR setup has actually been quite involved, eventually following a non-standard Reinforced Unlearning approach (QUARK) and modifying it with a dual-reward training scheme that addresses the two primary CTR constraints (coverage and faithfulness wrt the highlights). Indeed, the controlled decoding algorithm we adopted required only introducing a suitable reranking score. Analyzing its results, though, suggests interesting directions for future work on developing a better controlled decoding algorithm, deserving a dedicated research project. Overall, the methods we developed indeed achieve our original goal, yielding a high-performing CTR model, while setting the grounds for further research. \n\n***Question - training data quality***\n\nWe did in fact assess the quality of both training datasets in section 5.3 (lines 497-519), and showed that the GPT-4 generated data is of higher-quality.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) The proposed approach with three components is well motivated and reasonable. \n2) The approach achieves significant improvement over baselines. ", "reasons_to_reject": "1) Both the second and third components are borrowed from relevant studies with only slight changes.", "questions_to_authors": "Is there any way to evaluate the quality of original training data and the GPT-4 generated training data?", "paper_topic_and_main_contributions": "As a subtask of abstractive text summarization, controlled text reduction (CTR) is a recently introduced task which aims to generate a reduced and coherent text for a text with pre-selected content (i.e., highlights). This paper then proposes an approach by introducing three components. First, it uses GPT-4 to generate new summaries of the training data. Second, it fine-tunes the pre-trained model (i.e., the instruction-finetuned Flan-T5 large) with an existing RL but focuses on the source-side pre-selected context. Third, in inference it uses an existing controlled decoding algorithm but again focuses on the source-side pre-selected context. Experimental results demonstrate the effectiveness of the proposed approach. ", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "It lacks some description and statistics of the CTR dataset."}, "extra_text": {}}, {"review_id": "I7uQF5OCwU", "reviewer": "Reviewer_j7hm", "summary": "", "questions": "Question A: Why not test the two proposed strategies/GPT-4 distill data on the baseline's backbone model？\nQuestion B: See Reasons To Reject1", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the insightful comments, and would now address them one-by-one.\n\n***Issue with the limited contribution:***\n\nIn this paper, our main goal was developing a reliable controlled reduction model for downstream applications. This was motivated by the appealing potential applicability of this task in providing modularity and increased control in text generation, contrasted with the fact that the only available baseline model for the task had limited performance (~0.5 in both coverage and faithfulness). Since the main challenge in this task is to adhere to very tight constraints with respect to the input (covering strictly the highlighted content, faithfully and fully, while pulling only decontextualization information from the context), we opted to comprehensively explore relevant approaches for biasing models to adhere to the task constraints. This led us to cover the three approaches presented in the paper: symbolic knowledge distillation, RL and controlled decoding. While these general approaches are known, we had to tackle non-trivial challenges in order to apply them in the controlled reduction setting. In particular, the distillation phase required developing a high-performing chain-of-thought scheme, while the RL method required investigating different reward schemes, leading to a dual reward that addresses the two primary constraints of the task (coverage and faithfulness to highlights). While successful application of controlled decoding required less effort, altogether we had to address substantial challenges to come up with a high performing system (80’s-90’s), now enabling downstream usages.\n\nIn summary, while our paper is not focused on a particular algorithmic novelty, we view it as providing the following contributions: 1) comprehensive investigation of relevant strategies, including negative conclusions, for driving the CTR model to obey the task constraints; 2) non-trivial technical solutions, primarily regarding distillation and RL; and 3) a high-performing tool for the task for downstream use, thus achieving our targeted goal.\n\n***Issues regarding the combination of the two approaches (RL and controlled decoding):***\n\nWe thank the reviewer for raising the issues regarding method combination, in the second parts of comments 1 & 2. We now realize it deserves more attention. Driven by your comment, we performed the following, to be added to the paper:\n\n1) (per the 2nd part of issue #2) We conducted human evaluation also for the combination of the methods over our best performing backbone model (Flan T5 Distilled). The results are presented below, along with those of each method separately (as in the paper). We also add the F1 score for each line, to allow also a single-score comparison: \n| Algorithm           |    Faithfulness (P)    |     Coverage (R)    | F1|\n|-------------------|:----------------------:|:--------------------:|:--------:|\n| + RL                |           81.3         |          93.4       |   86.9   |\n| + Con. Decoding     |           85.6         |          91.3       |   88.3   |\n| Combined            |           83.4         |          92.3       |   87.7   | \n\nAs seen, these results are quite consistent with the corresponding behavior in the automatic evaluation (Table 1, which pertains to F1 scores). Overall, while the RL approach yields higher Recall and Con. Decoding yields higher Precision, the combined strategy yields some “midpoint” between the standalone methods, with a more balanced Recall-Precision tradeoff. Overall, in terms of F1 scores over the evaluated samples, the combination improves a little over the RL approach while performing slightly worse than the controlled decoding only, which is consistent with most automatic metrics as well. \n\n2) (per the 2nd part of issue #1) In addition, we performed a qualitative analysis, over the manually evaluated data, of the mismatching classifications between the combined model and each model alone. Notably, we found that for the RL method, the majority of False Positives, that is, content appearing in the summary which is not highlighted in the source, corresponds to non-highlighted spans within densely highlighted source contexts, where the highlighted and non-highlighted spans were syntactically related. In these False Positives, when generating a summary sentence, the RL method failed to exclude the non-highlighted content, which would have required some rephrasing. For instance, in the following example, the RL-based model failed to exclude the non-highlighted content, and extractively outputted the full source sentence (**bold** marks highlights): “**The new surveys led USDA experts to predict the 1988 corn crop will total** 4.48 billion bushels, **down** 37 percent **from last year's harvest** and the smallest output since 1983”. \\\n&nbsp; &nbsp;\nIn some of these cases, the incorporation of controlled decoding, which checks for highlighting adherence at inference time, succeeded in filtering out False Positive content. Yet, occasionally it omitted also the adjacent highlighted content, thus reducing coverage. This behavior points at the need to develop better controlled decoding algorithms than the one we leveraged in this work, which would be more successful in reranking beams that continuously preserve highlighted content while excluding non-highlighted information. Such an algorithm may better leverage the higher-quality beams generated by the RL model. Developing such a novel controlled decoding algorithm for inference time, which successfully couples with RL training, remains a challenge for future research. \\\n&nbsp; &nbsp;\nOverall, our paper provides a first thorough investigation of appealing approaches for imposing the strict biases of the controlled decoding task. Our findings indicate that each of RL, controlled decoding, and their combination currently yields a different faithfulness-coverage tradeoff balance, while calling for future work to develop a systematically superior combined approach.\n\n***Claim of insufficient experimentation with the backbone model*** \n\n(1st part of issue #2 and Question A)\nIf properly understood, we suspect that this comment might have arised due to some confusion or misses regarding certain information that does appear in the paper. We point at this information below, and will make sure in the final version to make it more prominent and augment it with details expected by the reviewer. \n\nAs mentioned in lines 435-441 (specifically 439-441), in our experiments, we compared the addition of our three contributions (distillation, RL, controlled decoding) over both the baseline model (LED) and our primary model (Flan T5), as backbones, testing all variants. As stated in these lines, LED was consistently inferior as backbone, in each configuration, relative to Flan T5, Consequently, and as stated, we included in Table 1 only the performance of the LED as the bare backbone model (1st line in the table), pointing at it’s inferiority relative to the bare Flan T5 (the second line), as we considered the figures of its consistently inferior performance in all configurations to be of lesser importance. Following the reviewer's comment, we will add this detailed information to the paper (possibly in an appendix), which will also show that the impact of our three contributions and their combinations over this model is consistent with their impact on Flan T5 (as was originally found in our experiments, but not mentioned in the paper).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Well-written paper. The paper is easy to follow.\n2. Extensive experimentation is conducted.", "reasons_to_reject": "1.  The contribution of the whole paper is limited.  The strategy of the proposed CTR model is almost derived from existing work, which greatly reduces the contribution of this paper. Even more baffling to me is the fact that experiments have shown that the two strategies (RL and Controlled Decoding) mentioned in the method decrease performance when used together, and the authors do not analyze this phenomenon in more depth. This makes me completely confused as to what the purpose of the proposed RL strategy is.\n2. The experimental part was grossly inadequate. The author only conducted experiments on Flan-T5, and the backbone of baseline compared with Flan-T5 was not Flan-T5 (it can be seen in Table 1 that the original FLAN-T5 had better performance than baseline). It is difficult to explain whether the effectiveness of this method is due to the selection of a strong backbone. At the same time, there are serious defects in the human evaluation experiment. The author did not evaluate the CTG model with two strategies for human evaluation. This is a very important point, which can rule out whether the lower score of the combination of the two strategies in the automatic evaluation is because the evaluation metrics are not good enough.", "questions_to_authors": "Question A: Why not test the two proposed strategies/GPT-4 distill data on the baseline's backbone model？\nQuestion B: See Reasons To Reject1", "paper_topic_and_main_contributions": "This paper focuses on the controlled text reduction task and processes a new CTR model.  The CTR model consists of two strategies: Highlights-Oriented RL Training and Highlights-Sensitive Decoding, and also  improve the quality of the CTR dataset by using GPT-4.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "N/A"}, "extra_text": {}}, {"review_id": "5VALXmjcm1", "reviewer": "Reviewer_vhDS", "summary": "", "questions": "1. Ad L267: Why is it important to alternate between rewards and not simply use a convex combination of $\\lambda r_1 + (1-\\lambda) r_2$?", "limitations": "", "rating": 4, "confidence": 1, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the appreciation of our work. \n\n***Main Concern - difficulty to follow the paper***\n\nThank you for bringing up this issue in a sincere manner. It is possible that the paper was organized for an audience familiar with our area. We will carefully check this aspect and provide additional background information and overall roadmaps to make the paper more easily accessible for broader readership.\n\n***Question***\n\nIn preliminary stages of our research, we did in fact experiment also with convex combinations of the two rewards in certain settings, and found them suboptimal compared to the alternating reward approach (which is also prominent in NLG for handling multiple rewards). Hence, we proceeded with the alternating reward scheme. Following your question, we performed additional experiments with convex combinations on our current models and settings, and found that these consistently lead to a decrease in performance across all metrics. We will add these results to the appendix, and will refer to them in the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed combination of method adjustements (RL+decoding) seems to be very potent at the task and work well. This could become a widely adapted approach for controlled text reduction.", "reasons_to_reject": "The paper was very difficult for me to parse, which can be attributed either to the writing organization or, more likely, my lack of expertise in this area.", "questions_to_authors": "1. Ad L267: Why is it important to alternate between rewards and not simply use a convex combination of $\\lambda r_1 + (1-\\lambda) r_2$?", "paper_topic_and_main_contributions": "The paper makes progress on the controlled text reduction task by using a better signal from the training data (highlights) during the training (reinforcement learning) and inference (constrained decoding).", "excitement": 4, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- According to title capitalization rules, even the second _\"don't\"_ in the title should be capitalized.\n- Figures 1 and 7 are included as raster images. They appear to be generated using matplotlib or some derivative thereof. You can use `plt.savefig(\"figname.pdf\")` before `plt.show()` to get the PDF (vector) version which you can include. This way the image will be of higher quality.\n- L215, L218: Use `\\log` instead of `log`. "}}]

Author rebuttal:

Meta-review: This paper studies the controlled text reduction task. The reviewers find the paper to be well-motivated, and attempt a combination of reasonable strategies. The paper also seems to benefit from good writing. However, the reviewers find the novelty of the work to be extremely limited with most of the techniques borrowed from the literature. The experiment section needs more work, with thorough human evaluation and clearly identifying the source of improvement. Given the importance of the task, this paper could establish baselines for the text reduction task in the future.
Decision: Accept-Findings