CTRLsum: Towards Generic Controllable Text Summarization
Authors: Junxian He
Wojciech Maciej Kryscinski
Bryan McCann
Nazneen Rajani
Caiming Xiong
Venue:  (2021)
Keywords: controllable text summarization

Abstract: Current summarization systems yield generic summaries that are disconnected from users' preferences and expectations. To address this limitation, we present CTRLsum, a novel framework for controllable summarization. Our approach enables users to control multiple aspects of generated summaries by interacting with the summarization system through textual input in the form of a set of keywords or descriptive prompts. Using a single unified model, CTRLsum is able to achieve a broad scope of summary manipulation at inference time without requiring additional human annotations or pre-defining a set of control aspects during training. We quantitatively demonstrate the effectiveness of our approach on three domains of summarization datasets and five control aspects: 1) entity-centric and 2) length-controllable summarization, 3) contribution summarization on scientific papers, 4) invention purpose summarization on patent filings, and 5) question-guided summarization on news articles in a reading comprehension setting. Moreover, when used in a standard, uncontrolled summarization setting, CTRLsum achieves state-of-the-art results on the CNN/DailyMail dataset.

Under review as a conference paper at ICLR 2021
CTRLSUM: TOWARDS GENERIC CONTROLLABLE TEXT SUMMARIZATION
Anonymous authors Paper under double-blind review
ABSTRACT
Current summarization systems yield generic summaries that are disconnected from users' preferences and expectations. To address this limitation, present CTRLsum, a novel framework for controllable summarization. Our approach enables users control multiple aspects of generated summaries by interacting with the summarization system through textual input in the form of a set of key- words or descriptive prompts. Using a single unified model, CTRLsum is able achieve broad scope of summary manipulation at inference time without requir- ing additional human annotations or pre-defining a set of control aspects during training. We quantitatively demonstrate the effectiveness of our approach on three domains of summarization datasets and five control aspects: 1) entity-centric and 2) length-controllable summarization, 3) contribution summarization pers, 4) invention purpose summarization on patent filings, and 5) question-guue -uuded summarization on news articles in a reading comprehension setting. Morcover, when used in a standard, uncontrolled summarization setting, CTRLsum achieves Mate-offthhearr results the CNN/DailyMail dataset
scientific pa-
INTRODUCTION
Neural summarization systems aim to compress document into short paragraph or sentence while preserving key information. There are largely two categories of summarization systems: extractive summarization that extracts important portions of a document (Cheng Lapata (016)] Nallapati et al. 2017 Narayan et al. (2018), and abstractive summarization that freely generates novel sentences (Rush et al. 2015 See et al. 2017; Paulus et al. 2018) which can produce coherent and fluent summaries more flexibly. In this paper we focus abstractive summarization.
Typically abstractive summarization methods take a document as input and yield a generic summary to cover certain information identified by the model. However, content of interest is user-dependent. Summaries should select information with respect to preferences of a user. For example, Figure [ shows an NBA basketball news article, and the reference summary describes several match results. However, fans certain basketball stars in these teams such as Lebron James or Stephen Curry might only be interested in the matches they played and would like know the player's scores as well.
Motivated by this, we focus on controllable summarization which allows the users to manipulate the summaries from the model. We propose CTRLsum, framework control summaries through control tokens in the form of a set of keywords or descriptive prompts. At training time, the model learns predict summaries conditioned on both the source document and keywords that serve as external guidance. During inference, keywords and optional prompts, which are the target prefix to constrain decoding, are combined as control tokens to convey user preferences as shown Figure
Keywords and prompts are complementary. Prompts do not perform well in many cases such as entity or length controlled summarization as our preliminary experiments imply, but keywords can achieve those goals in flexible way, for example, by using entity as keywords or varying the number of keywords to control entities and length respectively. However, keywords struggle in more open-ended scenarios like summarizing list of contributions of scientific papers, while constraining the de- coding with prompt "the main contribut ions of this paper are: (1) is possibly sufficient achieve the goal.
Code and model checkpoints will be public after the review period.

---

Under review as a conference paper at ICLR 2021
Dwyane Wade scored 21 of his 32 points the first half and Goran Dragic udded 20 as the Miami Hen handed LeBeon James anothur 106-92 victory
Keywords: Dwyaane Wade
Keywords-based Model
Miami beat Clovelind Cavaliens 106-92 at home on Menday. Dugane Wade scured 21 of points thu First half.
his over the Clevehad Cavaliers on Monday lignoring 60 solons] James soored 16 of his 26 points in the fourth quarter for Cleveland, which had four-game
streak Klay Thompson seored 26 points, and Stephen Curry had 19 points und nine assists as the playoff beaning depleted Angeles 108 105 ..... (gnorrig 400 tolens]
snapped. Kyrie Irving
Keywords
Keywords: Kyric Irving Lebron Janees
Golden State Warniors
Tagger
Control Tokens
Prompts
106-92 at home on Monday, Lebren Jomes scored 26 points but Kyrie Irving added 21.
Miami
Cleveland
Reference Summary
Miami Heat ended Cleveland Cavaliors
Keywords: Q: How many scores did Stephen Curry A:
with 106-92 Dallas
Oklahoma City 119-115. Golden State Warrions Toronto Raplors and Beston Coliãc alsu won.
(ipropp)) @
Stephion Curry ge? 4.]
Figure I: Workflow of the CTRLsum framework inference time. Users interact with summaries through textual control tokens the form of keywords or prompts. Keywords are required as input during training and testing, while prompts are optionally used at test time. Dashed lines represent optional paths control tokens can come from the source article, user, or both. The right portion of the figure shows actual outputs from CTRLsum.
CTRLsum is trained using only keywords as additional input which can be easily identified from training summaries. It requires neither extra human annotations pre-defining control aspects for training, yet is quite flexible to achieve a broad scope text manipulation as we will show this paper. In contrast, prior work primarily rely on pre-defined "control codes" (Fan al., 2018 Liu et al. 2018: Keskar et al. 2019). thus need to collect annotations for training and cannot generalize unseen control aspects easily at test time.
We use pretrained BART (Lewis et al. 2019) as the underlying architecture and perform experiments on three datasets in three distinct domains: CNN//ailymail news articles (Hermann al.) 2015))
arXiv scientific papers (Cohan et al., 2018), and BIGPATENT patent documents (Sharma et al.
2019). We quantitatively evaluate CTRLsum on five control aspects: (1) entity- ($4.2) and (2)
length-controllable summarization ($4.)), (3) summarizing the contributions of scientific papers, (4)
summarizing the purpose an invention ($4.4). and (5) summarizing answers to given questions in
zero-shot reading comprehension setting ($4.5). Notably, our approach also achieves comparable
or superior performance to the strong BART summarization model on all datasets in a standard,
uncontrolled setting ($4.6), leading to state-of-the-art results the CNN/Daillmmii dataset.
2
CTRLSUM
2.1 OVERVIEW
Unconstrained neural summarization methods trained to learn the conditional distribution p(y) (x). where x and y represent the source document and summary respectively. The generated summaries depend solely on the document x without human vnvolvement. To control the output summaries, we propose using additional control tokens z to represent user preferences and training a summarization model that predicts the conditional distribution p(y (x,xx z).
The control tokens include keywords as extra inputs during training and inference. They can also optionally include prompts test time further constrain the decoding process. As shown in Figure control tokens in the form of keywords, prompts, interface between users and an otherwise black-box neural model, providing flexible way for users to explicitly control automatic summarization. Next we describe how obtain automatic keywords for training as well as potential applications at test time.
combination of both act as an
2.2 AUTOMATIC KEYWORD EXTRACTION
In addition to extracting keywords from training data to train the model, CTRLsum also features automatic keywords extraction mechanism at test time, which can be used to suggest automatic
2

---

Under review as a conference paper at ICLR 2021
keywords according to user preferences, perform uncontrolled summarization without user signals. Next we describe the keywords extraction methods at training and inference time respectively.
Training. For training, we use the ground-truth summary to identify keywords in the source document. Specifically, we first greedily select sentences from the document that maximize the ROUGE scores (Lin 2004) with the reference summary. This step constrains keywords to those found in important sentences. Then, we identify all the longest sub-sequences in the extracted sentences that have matched sub-sequences in the ground-truth summary, similar to the copying word recognition method in (Gehrmann al. 2018). Finally, remove duplicate words and stop words and keep the remaining tokens as keywords. Compared to other keywords extraction methods (Riloff & Lehnert 1994] Mihalcea & 'Tarau 2004) which output only few salient words, our extraction retains most content words found in the summary. This encourages dependence on the given keywords by building a reliable correlation between their presence in the input and the target. It in turn ensures that user-provii rree-provide keywords are not ignored by the model at test time, which is catastrophic for controllable summarization system.
Inference. We formulate the keyword extraction problem test time sequence labeling task. Concretely, we train a BERT-based sequence tagger (Devlin et al. 2018) on the keywords and documents from training dataset. This tagger then computes the selection probability q, for each token in the test document. Similar to training time extraction, we first select n, sentences with the highest average token selection probability. Within these sentences words with gj > are selected as keywords up to maximum number of Maax.. The three hyperparameters Ng, Mmax are selected based on the uncontrolled summarization performance on validation datasets. The results are reasonably robust to different settings (see Appendix D for details).
2.3 SUMMARIZATION: TRAINING DETAILS
Format. At training time we prepend the keyword sequence to the source document separated with a special token. The summarization model is then trained to maximize p x, z) in an end-to-end they were in the source
fashion. The keyword sequence maintains the order of the keywords document, but we observe that the model often ignores this ordering as it frequently differs between source and target summary. We also separate keywords from different source sentences with the special token (")"). In applications where the sentence boundary is unknown, as when users propose their own keywords, the token can be ignored as in some of our experiments.
Keyword Dropout As mentioned in 222 our keyword extraction strategy retains most words from the summary found in the source document. Without regularization, the dependence on such keywords is strong enough that the model rarely generates novel words in the summary. To remedy this, we randomly drop keywords at training time so that the model leamn to rely on keywords that are present in the input, while also learning to still carry over key information from the source document that is not present in the keywords. Note that keywords dropout is applied at training time only.
Next going to introduce the five control aspects that we study in this paper as example use cases of sum.. Qualitative examples of them are shown in Table [
2.4 SUMMARIZATION: INFERENCE WITH KEYWORDS.
The keywords provide a generic interface to control multiple aspects of summaries, which allows the user optionally rely on automatically extracted keywords, user provided keywords, combination of both. This method provides clean separation of test-time user control and the training process, including pretraining. Consequently, CTRLsum can be adapted to new use cases without changing model parameters. For example, though nothing during training specifically focuses on controlling entities or length, examples below demonstrate the general applicability of keyword control to entity and length manipulation.
The goal of entity control is to produce summaries that focus on entities of interest. Figure I exemplifies summarization with respect different players when those player names are included as keywords directly influencing the summary.
Entity Control.
Length Control. Users may have different preferences as to the length of summaries. We allow such manipulation of the summary length through user-specified length parameter. Specifically,
3

---

Under review as a conference paper at ICLR 2021
Table 1: Qualitative examples from the output of CTRLsum. Left column shows source the generic reference summary. Keywords are bolded. "IJ" denote that the tokens are used as both keywords and prompts.
Source or Reference
Control Aspect
Keywords (bolded) or Prompts and Model Output
IS1S The reinforeements come four days after ISIS began atlacking Baiji oil relinery.
Source: Hundreds of additional Iraqi troops being sent In reinforce colleagues who are trying to fend off ISIS' attempt overunn Iraq's 's largest oil refinery, key paramilitary force said Tuesday. The reinforcements come four days after ISIS began altacking northern Iraq's 's Baiji oil refinery, a key strategic resource that has long been target because the facility refines much the fuel used by Iragis donnssticllyy. The additional troops came from Camp Speicher. a lottilied traui base near the city of Tikrit, according to the media office of the Hasd Al- Shaubi militia. The reinforcements include two rederal police regiments,
Hasd Al-Shaabi The reinforcements come from Camp Speicher, a fortified Iragi base near Tikrit. They include two federal police regiments, Iragi militury quick reaction force hattalion and a regiment from Hasd Al-Shaabi.
Eotity
(Length bucket 0) Iragi troops ISIS oil refinery come days attacking Baiji refinery base Tikrit- The re- inforeements come four days after ISIS began attacking Iraq's 's Buiji relinery. The additional troops came from Camp Speicher, a fortified Iraqi base near Tikrit.
Length
Iraqi military quick reaction force battalio lion and a regiment from Hasd Al-Shaabi. [ignoring 110 tokens] The refinery is kilometers (25 miles) from Tikrit.
[Q: Where did the additional troops Camp Speicher near the city of Tikrit.
from? A:]
QA
Reference: multiple selerosis is degencrative condition the central nervous system that affects nearly mil- lion of individuals in terms of their fignoring 60 tokens).
[the main contributions of this paper are: : )] propose regularization based data analysis pipeline for the prediction of the transition point between relapsing remitting and secondary progressive forms of multiple selerosis: ( ) we test the performance of the proposed pipeline on data collected from palients currenty enmilled in an ongoing funded project i (3) ) we discuss the potential role patient reported outcome measures proms _.) in the study of the disease evolution.
in work we resert regularized machine learning methods for binary classification and multiple output re- gression we propose a pipeline that can be used to predict the disease progression from patient reported the abtained madel is lested on a data set collected from ongoing clinical research project.
Contribution
Reference: an implantable intraocular pressure sensor system has a sealed goometric shape with an internal pres- sure value. sealed geometric shape has first light permitting surface and a second lexible surface.. pair of photocellss located a light shield is coupled to the second Rexible surface. when the second fiexible surface is deflecsed, a light mea- surement by the puir photocells indicates intraocu- lar pressure condition.
[the purpose of the present invention is] to provide an intraocular pressure measuring system that is small enough to be implanted in the of patient
sealed geometric shupe
Purpose
first separate the training data into 5 buckets by summary length so that each bucket has the same number of examples. Then we compute the average number of keywords K₁ for each bucket on the training data. At test time, user can specify length parameter E 1,, 2, 3, 4} to include the K keywords with the highest selection probability computed by the sequence tagger. This is similar to (Saito et al. 2020a), which uses the number of "guiding words" to control summary length.
2.5 SUMMARIZATION: INFERENCE WITH KEYWORDS AND PROMPTS
Prompts are pre-nef text sequences used as the target prefix to constrain decoding. They have been utilized to perform multi-purpose text generation with single unified model (Radford et al., 2019. Brown et al. 2020 In the CTRLsum framework, prompts are a kind of control token sequence, and we always use such tokens as both the target prefix and keywords (ablation results using prompts as keywords or prefix alone can be found in Appendix C.. We find that using prompts as keywords besides prefix helps focus on prompt-related content and mitigate the over wer-generation issue of vanilla summarization models, as we will show in $4.4 To the best of our knowledge, we are the first evaluate such prompt-based control method for summarization systems.
Summarizing Contributions. Existing datasets about scientific papers such as arXiv (Cohan et al. 2018) collect paper abstracts as the summaries, which often include extra background context and lack detailed contribution descriptions for the associated paper. In many cases, readers would benefit from an explicit list of contributions in order to understand the novelty and value of the paper. For these cases, we propose using control tokens "the main contri but ons of this paper are: (1) """ This prompt then triggers generation of a summary focused on contributions.
Summarizing Invention Purpose. Patent article summaries in existing datasets such as BIG- PATENT (Sharma et al.) 2019) can be over-complicated, often covering core method details. Yet for non-technical reader it would be preferred to provide a one-sentence summary that states the purpose of the invention while ignoring technical details. To apply CTRL in this scenario, use the
4

---

Under review as a conference paper at ICLR 2021
control tokens, "the purpose of the present invention is". This triggers concise summary focused on patent purpose.
Question-guided summarization. Human summarization can be constrained by questions (Krys) ciñski et al., (2019) that require answers to be found in the summary. This points to an important connection between summarization and reading comprehension that we further explore. We hypoth- esize that summarization model can directly answer some questions about the article if guided properly. This suggests the possibility of subsuming reading comprehension as a form of summariza- tion. To verify this hypothesis, reading comprehension behaviour.
the control tokens "Q: question text? A:" " to trigger
We note that prompts- and keywords- based control are complementary in practice while prompts could theoretically achieve any type of control, empirically they often do not work well for many aspects and the model is very sensitive to the precise wording of the prompt. For example, we found that using prompts such as "a summary focused on [entity) is:" or "a short s summary is:" does not work as well as explicitly using keywords for entity or length control (details can be found in Appendix (
3 RELATED WORK
Previous work on controllable summarization often collects control codes such as entity or length as supervision to train the model conditioned on both the code and article together (Fan et al.) [018]] Liu ct al. 2018 These methods do not generalize for controlling aspects of the summarization that were not seen during training. Recently (Saito et al. (2020a) use the number of word prototypes to control summary length in a similar way to how we use keywords. Interactive summarization provides a way for users to continuously control the information that is included in the summary (Bornstein et al.)
1999
Leuski et al., (000)). More broadly, controllable text generation has been studied for styles (Hu al. 2017 Fu al.) 2018 He et al., 2020b), topics (Tang et al. 2019 Huang et al. 2019). and
templates (Guu et al. 2018 Wiseman et al. 2018 He et al.. 2020a).
Keyword-guided text generation has been applied in other contexts with different motivations. Gehrmann et al. (2018) utilize copying words at test time to mask copying operations in sum- marization task. Li et al. (2018) and (Satoo al. (2020b) use keywords extra input to improve the uncontrolled summarization performance. Wang et al. (2016). Mou et al. (2016) and Yao et al. (2019) use textual input to plan poetry, dialogue, and stories respectively. wex.cclly-coostrraiiie decoding specifies certain lexicons as hard constraints in the target text (Hokamp & Liu| 2017 Post Vilar 2018). Prefix-constrained decoding was used machine translation (Knowles & Koehn 2016; Wuebker et al. 2016) and also demonstrate the multi-task ability present in large pretrained models (MoCann et al. 2018: Radford et al., 2019 Keskar et al. 2019 Brown et al. 2020).
EXPERIMENTS
Our experiments below are designed to (1) test the control efficacy of CTRLsum on five different aspects, and (2) examine the performance of CTRLsum in a traditional summarization setting without external control signals. Also, extensive model output examples can be found in Appendix E
4.1 EXPERIMENTAL DETAILS
We perform experiments on three distinct-domain summarization datasets: CNN//ailymail (CN- NDM) news articles (Hermann et al.] (015). arXiv scientific papers (Cohan et al. 2018). and BIGPATENT patent articles (Sharma et al. 2019). For all datasets the source documents are trun- cated 1024 tokens and the target summaries are truncated to 256 tokens following (Zhang et 2019). The conditional distribution p(y xx 2) in CTRLsum is our fine-tuned version of the pretrained LARGE model (Lewis et al., 2019). which achieves -att--of-theart performance on several summarization benchmarks. The automatic keyword tagger at test time is based on the pretrained BERTLARGE model (Devlin et al. 2018) fine-tuned as described in $2.2 Our summarization model implementation is based the fairseq toolkit (Ott et al.] 2019) and the automatic keyword extraction model is based on the HuggingFace Transformers library (Wolf et al., 2019). Complete setup and training details can be found in Appendix A.1
5

---

Under review as a conference paper at ICLR 2021
Table 2: Summarization performance with oracle entity length signals from the reference summary. "CTRL- sum (automatic)" represents our model using automatic keywords in an uncontrolled setting. LengthCode length-control baseline. Both BART and LengthCode numbers are from our runs.
CNNDM
arXiv
Model
R/U
BERTScore
ROUGE-1/21L
BERTScore
BART (Lewis et al. 2019
2224//1.25/41.00
0.336
15.16/17.36/400
0.164
CTRLsum (automatnc)
45.65/22.35/42.55
0,363
46.91/18.02/42.1
0.169
LengthCode (Fan et al. 2018
43.44/21. 10/40.35
0.346
45.91/17.33/41.38
0.147
CTRLsum (oracle entity)
48.75/25.98/45.4
0.422
CTRLsum (oracle length)
46.26/22.60/44 60/43.00
0.365
47.58/18.33/42.79
0.173
Table 3:
Entity control results CNNDM. Success rate is the fraction of decoded summaries that actually mention the given entity, while factual correctness the fraction of summaries that are judged as factually correct by human annotators. The BART numbers are in terms of unconstrained generated summaries. EntityCode numbers are directly from (Fan et al. 2018), which is obtained with a weaker convolutional seq2seq architecture and requires entity annotations at training time.
Success Rate (%) Full-article
Factual Correctness
Model
Lead-3
Important
Unimportant
BART (Lewis et al. 2019
61.4
29.0
98.0
EntityCode (Fan et al. 2018
61.2
33.8
CTRL.um
97.6
94.8
99.0
100.0
For evaluation, we measure commonly used ROUGE scores (Lin 2004) and the recently proposed BERTScore (Zhang et al. 2020) when ground-truth available. For control-related evaluation where often do not have reference summaries, (1) collect ground-trus summaries when possible, (2) examine whether summaries respect the control signal, or (3) resort to human evaluation.
4.2 ENTITY CONTROL
Setup. We first simulate user preference by providing the model with oracle entities extracted from the ground-truth target. Then we compare it to the model using automatic keywords in a uncontrolled setting show the effect of oracle entities. To examine whether the decoded summarics respect entity change, we sample 100 documents and repeatedly acquire every entity in the document to generate summaries, following Fan et al. (2188). Then compute Success Rate, the fraction of requested entity actually occurring in the output summaries. The results are reported in separation of whether the entity is from leading sentences from the full article. To test if the summaries from different entity input are factually consistent with the document, we sample another 100 documents, and for each we randomly sample one "important" entity that appears in the reference, and one "unimportant" entity that occurs neither in the reference nor the leading three source sentences to produce summaries. For each (article, summary) pair we ask annotators from Amazon Mechanical Turk to make a binary decision as to whether the summary can be entailed from the article. We then take the majority vote as the result and report the fraction factually correct summaries. We evaluate on CNNDM only since many examples in arXiv and BIGPATENT do not have identifiable entities.
Results. In Table we observe that the use of oracle entities helps boost the ROUGE-2 score by 3.6 points compared with using automatic keywords, which means CTRLsum is able take advantage of the given entities. Table shows the Success Rate and factual correctness evaluations. We include the numbers from |an et al. (2018) (EntityCode) for reference point. We note that their numbers come from convolutional seq architecture (see Appendix [ for ablation analysis on this) and their method utilizes entity annotations during training time, thus is not very comparable to CTRLsum. Remarkably, our model achieves high success rate for both lead-3 and full-article entities reaching around 95%. Yet other systems struggle to include the given entities especially for the ones that do not occur in the beginning of the article. Factual correctness scores from human annotators suggest that CTRLsum is able to generate factually consistent summaries no matter whether the entity of interest is important not, comparable to the unconstrained BART baseline.

---

Under review as a conference paper at ICLR 2021
Table 5: Fl scores the dev set of NewsQA and SQuAD, GPT2 results are from our runs. The BART baseline and GPT2 use prompts while CTRLsum use the same trigger as both keywords and prompts.
Table 4: Length control performance. MAD measures the deviation of output length from reference length, while PCC represents the correlation between given length signal and the actual output length.
Model
NewsQA
SQUAD
Model
MAD J
CNNDM
PCC T
MAD L
PCC
SpInBIERT Jeshi 2020
Supervised
73.0
94.6
BART
1.20
0.00
1.08
0.00
MatchL STM Wang
49.6
70.0
CTRLsum (automatic)
1.25
0.00
0.98
0.00
Zero-Shot
CTRLsum (-teggti
LongthCCod: Fun al. 2018
0.87
1.17
-0.02
0,53
0.69
1.06
0.00
0.48
Grrz-Larg (774M purams, who fine-luming)
BART (106M params. fine funed on CNNDM)
BARI (406M params, wie fine tuning)
12.6
24,9
8.2
41.7
23.5
15.8
CTRL sum (4IlM params, traised on CNNDM)
48.2
59.6
4.3 LENGTH CONTROL
Sctup. Similar entity control, we first examine the effect of oracle length signal from the reference to simulate user preference. In addition to ROUGE and BERTScore, we measure the length distance between the decoded summary and the reference following (Liu et al., 2018). Specifically, we compute the mean of absolute deviation (MAD) of the actual length bucket code lass of the decoded
summary from the ground-truth control code 4et, as * Σ> | ((m) |. To assess the summary variations as length signals change, we further sample 1000 documents and decode 5 different-lenghh summaries for each document. Then we report the Pearson Correlation Coefficient (PCC) between the input bucket code and actual bucket code. Experiments are conducted CNNDM and arXiv.
Results. In Table CTRLsum with oracle length signals only presents relatively small gains over the automatic CTRLsum baseline. This implies that oracle lengths only convey limited additional information to help generate the reference summary. We also the LengthCode baseline (Fan ct al.) 2018) based on BART, where the ground-trun length bucket code is prepended to the article at both training test time. However, LengthCode fails to consistently improve over BART with oracle length signals. Moreover, we find that the BART model fine-tuned with LengthCode method almost ignores the length signal with PCC close to O, as shown in Table This is not very surprising since length code would be less useful when the summarizers grow stronger, which can already learn a good length predictor implicitly. In contrast, CTRLsum with length-guided keywords achieves high positive PCC between control signal and actual output length, and is able to reduce the length deviation MAD compared to automatic baselines.
4.4 CONTRIBUTION AND PURPOSE SUMMARIZATION
Contribution Summarization Setup. There is existing dataset to evaluate contribution marization of scientific papers, bringing challenges to our evaluation. However, researchers often summarize the bullet contributions of their paper in the Introduction section, which inspire to extract such contribution claims as the reference summary. Therefore, we resort to the entire arXiv database and download all the papers whose first submission time is within the first six months of 2 that gives us 67K papers. We extract the Introduction section and bullet contributions with regular expression and filter out the ones that fail. The contributions are used as the reference and the Introduction section after removing the contribution claims is used as the source article we aim to predict contributions from the rest of the introduction section. This procedure leads 1018 test examples. We test the model trained on arXiv
Purpose Summarization Setup. To collect test dataset that features one-sentence invention purpose summaries, we sample 1000 test examples from BIGPATENT and present their reference summaries to human annotators from Amazon Mechanical Turk. For each example we ask one annotator to select the sentence that convey the purpose of the invention. We also provide the option for annotators that the invention purpose cannot be identified. After filtering out the invalid examples, we collect 763 examples as our test data.
We do not the arXiv set because we can only extract 20 valid test points from it. The entire arXiv datahase is at: https: : /7/www/ www. kaggle. com/ wwt./oom//caae CCrne Un vera wweerittyll teerritt//aiiiv
"The arXiv dataset used to train CTRLssum is collected before April 2018 according to their paper submission time, thus there should be data overlap hetween the training data and our contribution test data.
7

---

Under review as a conference paper at ICLR 2021
Table Summarization performance contributions of papers and purpose of inventions. The BART baseline uses prompts while CTRLsum use the same trigger as both keywords and prompts.
Contribution
Patent Purpose
Model
ROUGE-1/2/1
BERTScore (P/R/1I)
ROUGE-1/21
SeRe (P/R/FI)
BART (prompt)
43. 33.84/17.440 1...88447774/25... 74//2.889 43.88/18.17/22..1
141100.142/0.1330 130
29.05/11.8022
0.016/0.236/0.
CTRLsum (prompt*keywoom
0.179/0.098/0.138
1..4//1..77/4.24
11.800..52/0.16.
Table 7: Uncontrolled summarization performance. Automatic keywords from the sequence tagger, while oracle keywords are obtained utilizing the gold summaries. We report the oracle performance for a reference point. The BART results are from our runs. BS denotes BERTScore.
CNNDM
arv ROUGE- 1/2/L.
BIGPATENT
Model
ROUGE- 1/2/L
ROUGE- 1/2/L.
CTRLsum (Oracle Keywords)
64. 04.66//4042//60.2
0.555
56.08/25.0
1/500.23
0.268
55.19/26.62447.10
0.291
BART (Lewis al. 2019
44,24/21 225//1.06
0.336
15.16617..6/490
0.164
45.83/19.53/83
53/39.47
0.187
PEGAŠUS ZZhang 2019
44.70/17 27//5.2277//80
53.63/33.16/42..
CTRLsum (Automatic Keywords)
45.65/22.35/42.50
0.363
46.91/18.02/42.11
0.169
45.80/18.68/30/
0.188
Results. Table 6 shows results of contribution summarization on scientific papers and invention purpose summarization on patent filings. Through using the prompt text as both the decoder prefix and keywords. CTRLsum outperforms the BART baseline in most cases. We further report the precision (P) and recall (R) scores in BERTScore besides F1. We observe that the BART baseline tends over-verrett wver-generate a full summary with low precision scores while CTRLsum is able focus on keywords-related content.
4.5 QUESTION-GUIDED SUMMARIZATION
Setup. We directly test question-guided summarization on reading comprehension benchmarks in zero-shot setting. Specifically, we evaluate the CNNDM summarization models on in-domain NewsQA (Trischler al. 2017) and out-of-domain SQuAD 1.1 (Rajpurkar al. 2016) respectively. We note that some NewsQA test articles are present in the CNNDM summarization training dataset, yet we think it is still reasonable unsupervised setting since our model never sees questions or answers during training. In addition to comparing with the vanilla BART model, we also include the zero-shot performance from GPT2 language models (Radford et al.) 2019) (without fine-tuning) as a reference point. We omit the largest GPT2 model with 1.5B parameters since it cannot be evaluated in our single GPU device due to memory limits. We report Fl scores on the two benchmarks.
Results. BART is pretrained with a denoising task to predict the denoised version of the source, and performs poorly on zero-shot reading comprehension out of box, as shown in Table 5 Interestingly. however, BART fine-tuned on a summarization task without seeing any question-answer pairs in the training data - is able to improve the F1 scores by 24.4 and 25.9 points on NewsQA and SQuAD respectively. Moreover, CTRLsum equipped with question keywords is able to further boost the performance by 15.6 and 17.9 points, approaching the supervised MatchLSTM (Wang & Jiang (2117) score on NewsQA. Such results suggest that summarization might be a suitable transfer task for abstractive reading comprehension, which we leave for future work to explore.
4.6 AUTOMATIC SUMMARIZATION
Table shows the uncontrolled summarization performance without any user input, where our method uses the automatically extracted keywords as described in $2.2 On CNNDM and arXiv datasets CTRLsum outperforms the strong BART and PEGASUS baselines by a large margin, leading to new state-of-the-art performance CNNDM. It also performs comparably to the BART baseline on BIGPATENT in terms of BERTScore, though with an inferior ROUGE-2 score. Yet there is a big performance gap between BART-based models and PEGASUS BIGPATENT. The reasons might be different dataset processing El ssbboottimall learning schedule, or inherent difference between BART and PEGASUS.
8

---

Under review as a conference paper at ICLR 2021
Table 8: Human evaluation scores (scale 1-5, higher is better) entity control and purpose control experiments. Control accuracy (CA) and control relevance (CR) are reported. A score significantly different (according to the Welch Two Sample t-test, with < 0.05) than CTRLsum is denoted by *.
Important Entity
Unimpertant Entity
Purpose CA
Model
CA
CR
CA
CR
CR
CTRLsum
3.5
4.2
4.0
4.0
4.0
3.7
BART
3.8
3.7
1.3"
1.2
4,0
3.0"
Table 9: Human evaluation scores (scale 1-5, higher is better) of uncontrolled summarization performance.
Evaluation Dimensions from left to right are: factual consistency (FAC), relevance (REL), fluency (FLU), coherence (COH). A score significantly different (according to the Welch Two Sample t-test, with p < 0.05) than
CTRLsum (Automatic Keyword) is denoted by *
Model
CNNDM FACUREL/FLL//CO
arXiv FAC/RE//FLUUCO
BIGPATENT FAC/EL/FLUCCO
CTRLsum (Automutic Keyword)
4.6/4.644 1/4,1
4.1/4.//41/44 1/4,1
4.2/4.2/4.044.
BART
4.6/4.7/4.2/41
4.1/4.11/3.9/4..0
4,2/4.3/4.14
CTRLsum (Oracle Keyword)
1/4.1
4.2/4.3/4.0/4.1
4.244.//4.274.1
4.7 HUMAN EVALUATION
In this section we present human evaluation results for both controlled and uncontrolled summariza- tion. Full experiment details can be found in Appendix A.2
Controlled Summarization. We present further human evaluation results to evaluate "control" directly by informing annotators the intended control signal. We conduct experiments on entity and purpose control. Specifically, we inform the annotators our intent (to obtain summaries focused on a specific entity or purpose of patent), then we ask them to provide scores scale 1-5 over two dimensions: (1) Control Accuracy (CA): whether the summary contains accurate main information with respect to the intent, and (2) Control Relevance (CR): how the summary is relevant to the control intent overall summary that contains redundant contents that are unrelated to the intent will be penalized. Results including significance tests are shown in Table g The control accuracy for important entity control and purpose control are comparable between BART and CTRLsum without significant difference (p-value > 0.05), while CTRLsum shows significantly better control relevance overall by focusing on the desired information. Also, the unconstrained BART are unable generate niimporaan-eeniiy-relaaa summaries and thus suffers from poor scores on both dimensions.
Uncontrolled Summarization. We follow (Grusky et al. 2018 Fabbri et al. 2020) to ask human annotators from Amazon Mechanical Turk to score summaries (scale 1-5) over four dimensions: (1) Factual Consistency (FAC): the summary should only contain statements that can be entailed by the source document, (2) Relevance (REL): the summary should only contain important information of the source document, (3) Fluency (FLU): each sentence in the summary should be fluent, and (4) Coherence (COH): the summary should be well-structured and well-organized. Results including significance tests are present in Table The quality of summaries from all systems on all dimensions generally good with score mostly higher than 4.0. However, most scores do not show significant difference from CTRLsum (Automatic Keyword) with large p-values, despite their very different similarities against the reference summaries terms of ROUGE/BBRRSSoore (e.g. CTRLsum with oracle keywords). This implies that the summary quality from different systems powered by strong pretrained models like BART has become difficult to be clearly distinguished by non-expert MTurkers. We also note that non-expert human judgement for summarization may be unreliable and exhibit poor correlation with expert judgement (Gillick & Liu 2010 Fabbri et al. 2020).
5 CONCLUSION
In this paper we propose a generic framework to perform multi-aspect controllable summarization. The model is conditioned on keywords to predict summaries during training. At inference time the control tokens, in the form of keywords or prompts, enable users interact with models in a very flexible way. Experiments five different control aspects demonstrate the efficacy of our method.
*PEGASUS updated the BIGPATENT data preserve casing and applicd some format cleaning.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Jeremy Bornstein, Douglass R Cutting, John D Hatton, and Daniel E Rose. Interactive document summarization, 1999. US Patent 5,867.164.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020.
Jianpeng Cheng and Mirella Lapata. Neural summarization by extracting sentences and words. In Proceedings of ACL, 2016.
Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. A discourse-aware attention model for abstractive summarization of long documents. In Proceedings of (AACL((Shor Papers), 2018.
Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. 2018.
Alexander R Fabbri, Wojeiech Kryšciński, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. Summeval: Re-evaluating summarization evaluation. arXiv preprint arXiv:2007. 11177.12220 /2626, 2020.
Angela Fan, David Grangier, and Michael Auli. Controllable abstractive summarization. In Proceed- ings of the 2nd Workshop on Neural Machine Translation and Generation, 2018.
Zhenxin Fu, Xiaoye Tan, Nanyun Peng, Dongyan Zhao, and Rui Yan. Style transfer in text: Explo- ration and evaluation. 2018 8.
Jonas Gehring, Michael Auli, David Grangier. Denis Yarats, and Yann N Dauphin. Convolutional sequence sequence learning. In Proceedings of ICML. 2017.
Sebastian Gehrmann, Yuntian Deng, and Alexander M Rush. Bottom-up abstractive summarization. In Proceedings of EMNLP, 2018.
Dan Gillick and Yang Liu. Non-expert evaluation of summarization systems is risky. In Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon Mechanical Turk, 2010.
Max Grusky, Mor Naaman, and Yoav Artzi. Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies. In NAACL, 2018
Kelvin Guu, Tatsunori Hashimoto, Yonatan Oren, and Percy Liang. Generating sentences by editing prototypes. Transactions of the Association for Computational Linguistics, 6:437-450, 2018.
Junxian He, Taylor Berg-Kirkpatrick, and Graham Neubig, Learning sparse prototypes for text generation. In Proceedings of NeurlPS, 2020a.
Junxian He, Xinyi Wang, Graham Neubig, and Taylor of unsupervised text style transfer. In Proceedings of ICLR, 2020b.
A probabilistic formulation
Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In Proceedings of NeurlPS, 2015.
Chris Hokamp and Qun Liu. Lexically constrained decoding for sequence generation using grid beam search. In Proceedings of ACL, 2017.
Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric Xing. Toward controlled generation of text. In Proceedings of ICML. 2017.
Qiuyuan Huang, Zhe Gan, Asli Celikyilmaz, Dapeng Wu, Jianfeng Wang, and Xiaodong He. Hi- crarchically structured reinforcement learning for topically coherent visual story generation. In Proceedings of AAAI, 2019.
10

---

Under review as a conference paper at ICLR 2021
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S Weld, Luke Zettlemoyer, and Omer Levy. Spanbert: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 8:64-77, 2020.
Nitish Shirish Keskar, Bryan McCann, Lav R Varshney, Caiming Xiong, and Richard Socher. Ctrl: A conditional transformer language model for controllable generation. arXiv preprint arXiv:/909.00 05858, 2019.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proceedings of ICLR, 2015.
Rebecca Knowles and Philipp Koehn. Neural interactive translation prediction. In Proceedings of the Association for Machine Translation in the Americas, pp. 107-120, 2016.
Wojeiech Kryšciííski, Nitish Shirish Keskar, Bryan McCann, Caiming Xiong, and Richard Socher. Neural text summarization: A critical evaluation. In Proceedings of EMNLP, 2019. 2019.
Anton Leuski, Chin- Yew Lin, and Eduard Hovy. ineats: interactive multi-document summarization. In Proceedings of ACL, 2003.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-te cectnncett--squuuee to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13400 2019.
Chenliang Li. Weiran Xu, Si Li, and Sheng Gao. Guiding generation for abstractive text summariza- tion based on key information guide network. In NAACL (Short Papers), 2018.
Chin- Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, 2004.
Yizhu Liu, Zhiyi Luo, and Kenny Zhu. Controlling length in abstractive summarization using convolutional neural network. In Proceedings of EMNLP, 2018.
Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. The natural language decathlon: Multitask learning as question answering. arXiv preprint
08730, 2018.
Rada Mihalcea and Paul Tarau. TextRank: Bringing order into text. In Proceedings of EMNLP, 2004.
Lili Mou, Yiping Song, Rui Yan, Ge Li, Lu Zhang, and Zhi Jin. Sequence to backward and forward sequences: A content-introducing approach to generative short-text conversation. In Proceedings of COLING, 2016.
Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. SummaRuNNer: a recurrent neural network based sequence model for extractive summarization of documents. In Proceedings of AAAI, 2017.
Shashi Narayan, Shay B Cohen, and Mirella Lapata. Ranking sentences for extractive summarization with reinforcement learning. In Proceedings of NAACL, 2018.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL (Demo Paper), 2019.
Romain Paulus, Caiming Xiong, and Richard Socher. deep reinforced model for abstractive summarization. In Proceedings of ICLR, 2018
Matt Post and David Vilar. Fast lexically constrained decoding with dynamic beam allocation for neural machine translation. In Proceedings of NAACL, 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl Blog, 18))9, 2019.
Pranav Rajpurkar, Jian Zhang. Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of EMNLP, 2016.
11

---

Under review as a conference paper at ICLR 2021
Ellen Riloff and Wendy Lehnert. Information extraction a basis for peeeh-reeisiion text classification. ACM Transactions on Information Systems (TOIS), 12(3):296-3 -333, 1994.
Alexander M Rush, Sumit Chopra, and Jason Weston. A neural attention model for abstractive sentence summarization. In Proceedings of EMNLP, 2015.
Itsumi Saito, Kyosuke Nishida, Kosuke Nishida, Atsushi Otsuka, Hisako Asano, Junji Tomita, Hiroyuki Shindo, and Yuji Matsumoto. Length-controllable abstractive summarization by guiding with summary prototype. arXiv preprint arXiv:2001.07331, 2020a.
Itsumi Saito, Kyosuke Nishida, Kosuke Nishida, and Junji Tomita. Abstractive summarization and saliency models. arXiv preprint
with combination of pre-trained arXiv:2003.13008, 2020b.
Abigail See, Peter J Liu, and Christopher D Manning. Get to the point: Summarization with pointer-generator networks. In Proceedings of ACL, 2017.
Eva Sharma, Chen Li, and Lu Wang. BIGPATENT: large-scale dataset for abstractive and coherent summarization. In Proceedings of ACI,
Jianheng Tang, Tiancheng Zhao, Chenyan Xiong. Xiaodan Liang, Eric Xing, and Zhiting Hu. Target- guided open-domain conversation. In Proceedings of ACL, 2019.
Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. Newsqa: A machine comprehension dataset. In Proceedings of the 2nd Workshop on Representation Learning for NLP, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In Proceedings of NeurIPS, 2017.
Daisy Zhe Wang. Wei He, Hua Wu, Haiyang Wu, Wei Li, Haifeng Wang, and Enhong Chen. Chinese poetry generation with planning based neural network. In Proceedings of COLING, 2016.
Shuohang Wang and Jing Jiang. Machine comprehension using match-lstm and answer pointer. In Proceedings of ICLR, 2017.
Sam Wiseman, Stuart M Shieber, and Alexander M Rush. Learning neural templates for text generation. Proceedings of EMNLP, 2018.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Huggingface's 's transformers: State-of-the-aa natural language processing. ArXiv, abs/1910.03771, 2019.
Joern Wuebker, Spence Green, John DeNero, Saša Hasan, and Minh- Minh-Thang Luong. Models and inference for mrefix-constraieed machine translation. In Proceedings of ACL, 2016.
Lili Yao, Nanyun Peng. Ralph Weischedel, Kevin Knight, Dongyan Zhao, and Rui Yan. Plan-and- write: Towards better automatic storytelling. In Proceedings of AAAI, 2019.
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. arXiv preprint arXiv: 192.08877 2019.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. BERTScore: Evaluating text generation with bert. In Proceedings of ICLR, 2020.
12

---

Under review as a conference paper at ICLR 2021
A EXPERIMENTAL SETUP DETAILS
A.1 GENERAL SETUP
In this section we include additional experimental details left out in the main content due to space limitations. We fine-tune the pretrained MARTLARGE model all our experiments. Specifically we use the bart large checkpoint from fairseq (Ott et al., 2 For all BART-based summarization models, we fine-tune with learning rate 3e-5 and a polynomial learning rate decay schedule, the optimizer is Adam (Kingma & Ba 2015) and batch size is 64. Our optimization scheme and hyperpa- rameters follow the BART fine-tuning instructions in fairseq examples. We train the summarization models with 20k steps on CNNDM, 50k steps on arXiv, and 300k steps on BIGPATENT. We train the BERT tagger with learning rate 5e-5, Adam optimizer, and batch size of 128 on all datasets. Similar summarization models, the tagger trained with 20k, 50k, and 300k steps on CNNDM, Xiviv, and BIGPATENT respectively. Also, we adopt sliding window approach so that the BERT-based tagger is able to handle sequences that are longer than 512 tokens. For both ROUGE and BERTScore evaluation, we report the F1 measure. We report the rescaled BERTS and the hash code is roberta-large__ __________e_ ceri-nooiioo ltler_ooodd___eriiiiiieee www.eeerionnsoee 3.6 6 (hug_trans=3) 0. -resca led.
As mentioned in 22 we need three hyperparameters for automatic keywords extraction during inference the number of pre-selected sentences n,s the selection probability threshold e, and the maximum number of keywords Mmmax. We select these hyperparameters for each dataset based on the uncontrolled summarization ROUGE-2 score on validation dataset. The summarization performance is robust these hyperparameters in reasonable range, shown in Appendix D Specifically, use {n, = 10, e = 0.25, maax = 30} for CNNDM, {n. = 10, e = 0.15, mmax = 40) for arXiv, and (n.- = 5, < 0.15, mmax 30).
Invention Purpose Summarization. In the experiment of summarizing invention purpose on patent articles ($4.4). We examined whether the model would possibly copy source sentences through matching the prompts, we search strings in the form of "the purpose of [some words or phrases] is" among 763 test examples, and only test articles are identified. This means the models are not generating by exactly matching prompts most of the time.
A.2 HUMAN EVALUATION SETUP
Here we include details about human evaluation experiments in $4.7
Controlled Summarization. For controlled summarization, we sample 100 examples for each task, and summaries of each example from all systems are presented together to the human annotator to be scored. For CNNDM we provide article and summaries, while for BIGPATENT we provide reference and summaries using the reference summary because the source patent documents are very long and hard to be read by non-expert humans. We did not evaluate contribution summarization since it is unrealistic to ask humans judge contributions of many scientific papers from various domains. We tried to hire workers from Amazon Mechanical Turk first, but we failed obtain reliable results from them they often ignored the given user intent and tended to score the text as uncontrolled summaries (reflected by very poor scores on unimportant-entig summaries because these summaries do not contain the main information of the article), even though we instructed them that the control signal is critical. Therefore, we ask two independent human annotators through personal correspondence from the authors of this paper. One of the annotator is PhD researcher physics, and the other is a law graduate on intellectual property in the United States. They are able to follow the given control intent and considered more reliable than the MTurkers. We take the average of two annotators as the score for each example, and average over all examples to obtain the final score.
surrogate for the source article. This is
Uncontrolled Summarization. For uncontrolled summarization, we sample 100 examples for each dataset, and hire 3 independent workers from Amazon Mechanical Turk to conduct evaluation. For CNNDM we provide article and summaries, while for arXiv and BIGPATENT we provide reference and summaries using the reference summary source patent documents or scientific papers are very long and hard to be read by non-expert humans. Summaries of each example from all systems are presented together to the human annotator to be
surrogate for the source article. This is because the
13

---

Under review as a conference paper at ICLR 2021
scored. The median score of 3 workers is taken for each example, and average over all examples is reported.
B ABLATION ANALYSIS OF ENTITY CONTROL
In Table 3 observe that CTRLsum achieves very high success rate (~ 95%) of entity control,
compared to previous work (Fan et al., 2018) which can only succeed 61.2% and 33.8% of the time lead-3 and full-article entities respectively. We perform ablation analysis understand the important gradients that contribute to the success of CTRLsum. We train CTRLsum with another two architectures in addition to BART: (1) convolutional seq2seq (Gehring al., 2017) with the same hyperparameters as in (Fan et al. 2018), and (2) transformer seq2seq with the same hyperparameters the base model in (Vaswani al. 2017). Note that the transformer model is trained from scratch without pretraining. Results are shown in Table IO CTRLsum parameterized with weaker convolutional seq2seq architecture fails to depend on the keywords well with an over 40-point success rate drop, yet the success rate of transformer seq2seq without pretraining only drops around 5 points. This implies that the transformer seq2seq architecture is critical for CTRLsum to depend on the keywords well, while pretraining can further improves it
Table 10: Entity control results CNNDM. Success rate is the fraction of decoded summaries that actually mention the given entity.
Success Rate (%) Full-article
Model
Lead-3
BART (Lewis et al. 2019
61.4
29.0
Fan 2018)
61.2
33.8
CTRLsum (Cony Seq2Seg)
50.1
23.3
CTRLsum (Transformer Seq2Seq)
92.6
88.3
CTRLsum (BART)
97.6
94.8
C ABLATION ANALYSIS ON KEYWORDS AND PROMPTS
In the controlling aspects we studied CTRLsum uses control tokens either as keywords alone (entity and length), as keywords and prompts together (contribution, purpose. QA). Here we present further results when control tokens are used as prompts, keywords, or both for entity control, contribution control, and NewsQA tasks. Specifically for entity control, we use the control tokens "a summary focused on [entity] is:" for "prompt" and "prompt + keyword" variants In this case success rate is computed excluding the prompt text. The control tokens for other settings the same as previous experiments. Results are shown in Table [1 where keywords and prompts are of different importance for different tasks and complementary in general. For example, using prompts to control entities turns out to be difficult with a very low success rate - we find that the system fails understand the prompt and produce summaries appropriately in most cases. However, prompts contribute the most to contribution summarization with comparable performance with using prompts and keywords together, while removing prompts and using keywords alone suffers from drastic performance drop to trigger the contribution. For NewsQA task, prompts and keywords demonstrate mixing effectiveness using either of them alone experiences over 20 points loss compared to using them together.
'For reference points, the ROUGE- 1/2/L (with automatic keywords) of CTRLsum (Conv Seq2Seq) is 41. /9919.71//88.... while CTRLsum (Transformer Seq2Seq) obtained 43.69/20.78/40.55.
'We tried several prompts variants, for example, QA style ones "Q: What happened to [entity] ] A:" "Q: what do we know about [entity]? A:". None of them lead to meaningful entity control.
14

---

Under review as a conference paper at ICLR 2021
Table 11: Ablation analysis on the role of keyword and prompt respectively. Entity success rate refers the full
article entity success rate.
Entity Success Rate (%)
Contribution
NewsQA F1
Model
ROUGE-1/2/L
BERTScore
CTRLsum (keyword)
94.8
39.96/12.74/22.66
0.088
15.5
CTRLsum (prompt)
17.6
00.82/18.12/11
664
0.133
26.3
CTRLsum (prompt keyword)
12.6
43.88/18.17/27.79
0.138
48.2
15

---

Under review as a conference paper at ICLR 2021
D ROBUSTNESS ANALYSIS OF KEYWORDS EXTRACTION HYPERPARAMETERS
Table shows the ROUGE-2 scores of uncontrolled summarization on the validation set with different keywords extraction hyperparameters. We more fine-grained stride size to iterate the mmax hyperparameter for CNNDM since its source articles are usually shorter than arXiv and BIGPATENT. As observed, the automatic summarization performance relatively robust these hyperparameters in reasonable range.
Table 12: ROUGE-2 scores of uncontrolled summarization on the validation set with different keywords extraction hyperparameters.
Model
CNNDM
arXiv
BIGPATENT
0.10, f2 5, Mmax
25
22.82
C 0.10, 72 5, Mmax
30
22.71
17.81
18.60
0.10, fl 5, Mmmax 35
22.54
e < 0.10, f, 5, Mmmax 40
17.96
18.35
e 0.10, n, 10, mamx 25
22.83
E 0.10,
10, Mlax = 30
22.67
17.99
18.61
E 0.10, n 10, Mmax 35
22.44
c <= 0.10, 10, Maax = 40
18.03
18.04
c 0.15, n 5, Mmaxx 25
22.85
0.15, 5, 10max 30
22.79
17.80
18.79
6 0.15, f1 5, 1lmax 35
22.71
0.15, f2 5, Mmax 40
17.95
18.76
E = 0.15, 71 10, Maax 25
22.85
0.15, f 10, Maax 30
22.77
17.99
18.76
C = 0.15, ng 10, Mmax 35
22.41
0.15, n 10, maax
40
18.05
18.62
6 0.20,
5, Mmax
25
22.86
e 0.20, n1
5, 10max 30
22.87
17.71
18.77
с 0.20,
5, Mmax 35
22.89
E 0.20, n 5, Mmax 40
17.88
18.71
0.20, 10, mama 25
22.87
6 0.20, n 10, mmax 30
22.85
17.88
18.77
0.20, r 10, maax 35
22.84
E 0.20, n 10, Mmax 40
17.98
18.73
0.25, f
5, Mmmax
25
22.84
c = 0.25, n2 5, Mmax 30
22.88
17.57
18.67
0.25, 7
5, Mmax 35
22.91
6 0.25,
5, Mmmax 40
17.71
18.66
0.25, f4 10, Almax 25
22.90
E 0.25, = 10, Maax = 30
22.95
17.76
18.72
E 0.25, n, 10, Mmax 35
22.95
0.25, 10, Mmax 40
17.84
18.70
c 0.30, n 5, Mmaxx 25
22.58
c 0.30, 5, Mmax 30
22.62
17.24
18.53
6 0.30, nl 5, 7mmax 35
22.63
0.30, f1 5, 70max 40
17.32
18.52
E 0.30, n, 10, Mmax
25
22,65
0.30, n 10, maxx
30
22.70
17.38
18.55
6 0.30, n, 10, Maxx = 35
22.70
0.30, ng 10, Manxx
40
17.44
18.55
16

---

Under review as a conference paper at ICLR 2021
E RANDOM OUTPUT EXAMPLES
In this section, we randomly sample test examples and show the source aticle, reference summary, and the model ouiput from CTRLsum for each control aspect.
E. ENTITY CONTROL
For entity control, we randomly sample articles from CNNDM and for each article we randomly select 5 entites as keywords to show the model output.
Table 13: Random Entity Control Examples
Americans on the United States' no-fly list will now be privy to information about why they have been bunned from commercial flights and be given the opportunity dispute their status, according to court documents filed by the Justice Department this week. The revised policy comes in response to a June ruling by federal judge that suid the old process was in violation of the Fifth Amendment's 's guarantee of due process. The decision was part of an American Civil Liberties Union lawsuit brought on behalf of 13 Americans on the list. But the ACLU isn't satisficd with the govemment's 's new policy, outlined documents filed Monday federal courts Oregon (PDF) and Virginia (PDF). "After years of fighting in court for complete secrecy and losing, it's good that the government is finally now going to tell people of their status on the No Fly List." said Hina Shamsi, director of the ACLU National Security Project and the lead attorney on the case, in a statement. "Unfortunatlyy we've found that the govermment's new redress process falls far short constituional requirements because it denies our clients meaningful notice. evidence, and a hearing. The government had an opportunity come up with fair process but failed, so we're re challenging it in court again." People on the no-fly list, managed by the FBI's Terroistt Screening Center, are prohihited from boarding a commercial flight for travel into or out of the United States. The number of people on the list classified. An official with knowledge of the government's 's figures told CNN in 2012 that the list contained about 21,000 names, including about 500 Americans. Before the change, American citizens and permanent residents who inquired with the government about heing denied aireraft boarding received a letter that neither confirmed nor denied their inclusion on the no-fly list. Now, they IIl be made aware their status they apply for redress, with an option request further information. 'The U.S. government is making enhancements the Department of Homeland Security Traveler Redress Inquiry Program (DHS TRIP) provide additional transparency and process for U.S citizens and lawful permunent residents who have been denied hoarding on a commercial aircraft because they are on the No Fly List." the Department of Homeland Security said in statement. In cases which travelers included on the request to receive detailed response,
Article
submit more information about their status, the govermment will provide second, identifying "specific criterion under which the individual hus been placed on the No Fly List." " according to the court documents, An unclassified summary of that information will be provided "to the extent feasible, consistent with the national security and law enforvement interests at stuke, court papers said. Those who appeur the no-fly list will then have further opportunity to dispute their status writing. with supporting materials or exhibits, and will receive final written decision from the Transportation Security Administration. The 2014 ruling that prompted the policy chunges had called for passengers on the list to be given the opportunity to dispute their status before judge.
Reference Summary
Americans on the no-fly list will now get info about why they've 've been banned from flights. ACLU says the policy still denies "meaningful notice, evidence, and a hearing"
the Fifth Amendment's: The new policy is in response June ruling that said the old process was in violation of the Fifth Amendment's 's guarantee of due process.
Civil Liberties Union: The American Civil Liberties Union says it's not satisfied with govern-
ment's 's new policy.
the Transportation Security Administration: The Transportation Security Administration will pro- vide travelers with more information about their status. A June ruling had called for passengers on the list to given the opportunity to dispute their status before judge.
CTRL
Virginia: The ACLU is challenging the new policy in federal courts in Oregon and Virginia.
FBI: People the no-fly list, managed by the FBI's Temorist Screening Center, prohibited from boarding commercial fligtts. The ACLU isn't 't satisfied with the guvernment's i's new policy.
17

---

Under review as a conference paper at ICLR 2021
Table 14: Random Entity Control Examples
Point guard Stephen Curry nearly single-handedly outscored New Orleans with 11 first quarter points as the Warriors built a p0-pti lead and rolled to victory in Game One of their Western Conference first-round series, Gume Two in the best-of-seven series is scheduled for Monday night in Oakland. Golden State, the top seed in left
regular season, recording a 19th straight home win and 40th 42 games this year. Stephen Curry scored stunning 34 points for the Golden State Warriors there play-off game. The Warriors did it by tuking 25-point lead into the final minute of the third quarter, then holding "We missed a lot of free throws, which made it lot closer than it needed to be. couch Steve Kerr suid. 'But in playofl's you' ve just got to get it done somehow. We's up 1-0. Warriors with 34 points, hitting 13 of 25 shots and four three-pointers. All five Golden State starters scored in double figures. Guard Klay Thompson complemented Curry with 21 points, while power forward Druymond Green (15 points, 12 rebounds) and center Andrew Bugut (12 points, 14 rebounds) recorded double-doubler The point guard has been in spectacular form he looks to lead Warriors the NBA glory. Curry celebrates after scoring three-pointer
West, picked up right
where we want
Curry the
Atticle
his way scoring first quarter points. New Orleans power forward Anthony Davis scored game-high high 35 points. 20 in the fourth quarter. He also blocked four shots and grabbed seven rebounds. Small forward Quincy Pondexter, assigned to cover Curry most of the game, added 20 points for the Pelicans. Visiting Washington outscored Teronto 11-4 the overtime in winning the opener of their first-round playoff series. Washington traveled Toronto and came away with the victory with confident overtime perfornance. Forward Paul Pierce started the overtime with three-pointer and scored 20 points to lead the Wizards, who let a 15-point lead slip away in the fourth quarter before winning. Guard Bradley Beal added 16 points Washington and forward Nene contributed 12 points and 13 rebounds. Forward Amir Johnson came off bench lead the Raptors (left) top-scored in the clash with 20 points the Wizards secured an important play-off win.
points and guard DeMar DeRozan added 15 points.. Paul Pierce
Stephen Curry scored 34 points for Golden State against New Orleans. The Californian- team defeated the Pelicans 106-99. Washington Wizurds outscored the Toronto Raptors 11-4 overtime. Paul Pierce led the scoring with 20 points for the Wizards.
Reference Summary
Stephen Curry: Stephen Curry scored 34 points us Golden State Warriors beat New Orleans Pelicans.
Washington Wizards beat Toronto Raptors overtime play-off' game.
Oakland: Stephen Curry scored 34 points for the Golden State Warriors. Game Two in the best-of- seven scries scheduled for Monday night Oukland.
Steve Kerr: Stephen Curry scored 34 points as the Warriors beat the Pelicans. Coach Steve Kerr said: 'We missed a lot of free throws, which made it a lot closer than it needed to be"
CTRLsum
Klay Thompson: Stephen Curry scored points as Warriors beat the Pelicans. Klay Thompson
and Draymand Green also scored in double figures.
Andrew Bogut: Stephen Curry scored 34 points as the Warriors beat the Pelicans. Draymond Green and Andrew Rogut both recorded double-
18

---

Under review as a conference paper at ICLR 2021
Table 15: Random Entity Control Examples
It's the ultimate treat for Benedict Cumberbatch fans and stands an imposing 6ft tall just like the man himself. But shoppers at London's Westfield Stratford City shopping centre looked more than little surprised to discover a chocolate sculpture of Benedict Cumberbatch in their midst. One lady was spoited cautiously approuching the edible artwork hefore quickly hacking off, while another couldn't quite for video. Finishing touches: The sculpture is readied for its big unveiling at Westfield Stratford City shopping centre. Oh dear: Reaction to the sculpture was mixed, with some shoppers bursting into laughter. Even less impressed was the shopper who stood stony-faced in front of the creation for several moments, while another burst into laughter as soon as she spotted it. did, however, prove an immediate hit with puir of police sniffer dogs who wagged their tuils as they gave thorough snifting down. The artwork, which hus been given pride of place in the shopping mall's atrium, was commissioned by UKTV to mark celebrate its screening of the third series of Sherlock. It took a crew of eight people to complete the sculpture, which look over 250 mun hours to create and weighs 40kg. Does look like me? Benedict Cumberbutch strikes a pose with James Corden during Oscars party. Mixed reaction: pair of police sniffer dogs loved the sculpture but shoppers looked baffled. Hilarious: A lady bursts into laughter after spotting the 6ft homage to Mr Cumberbatch. Not amused: A shopper looks thoroughly unimpressed us she contemplates the artwork. Luckily for Cumberbutch. who usually enjoys considerably more complimentary response to projects he's involved in. the piece will only be residence temporarily. The 38-ycar-old actor, who currently expecting his first child with wife Sophie Hunter, 37. isn't the only famous face to have found himself the subject of an edible artwork. In the up to the release of 50 Shades of Grey, bakers created not one but two 6ft gateaux paying homage to Jamie Doman. One depicted the actor the grey suit beloved of his Shades character Christian Grey, while the other showed him topless and came complete with an edible six pack. Award Award-winning: Both Jennifer Lawrence and her cake alter- alter-ego have won awards. Homage: The cake, which triumphed show last November. was inspired by the Hunger Games. Actress Jennifer Lawrence has also been immortalised in cake, with baker Lara Clarke creating a sweet treat designed to resemble the 24- yeay-ro-s'd's Hunger Games alter-ego, Katniss Everdeen. The confection, which was baked ahead
their smile of surprise. Semoll down
Atticle
release Mockingjay Part One November, met with the approval Lawrence herself, who, when asked about said Ms Clarke was 'incredibly talented'. Other A-listers to get the culinary treatment include Kevin Bacon, whose likeness was made from bacon, and Taylor Swift who featured on a cake. The Duke and Duchess of Cambridge were turned into losty-looking pizza, while President Barack Obama had his likeness sculpted from butter. immortalised: The Duke and Duchess Cambridge have been tumned into pizza the past. ribute: Kevin Bacon has a bacon sculpture his name while Jamie Doman has heen made into cake.
A 6ft chocolate sculpture of Benedict Cumberbatch has been unveiled. Toothsome statue has been placed inside London shopping centre. But shoppers reactions the creations were decidedly unenthusiastic. One woman glared at while others just looked thoroughly baffled. It did manage to win the approval of pair of police sniffer dogs. It weighs 40kg and took cight people 250 man hours to create. Other celebritics to get culinary tributes include Jennifer Lawrence, Her 6it cake won an award- and the 24 rear-or--'d's 's approval. Actor Kevin Bacon has also been immortalised in hacon.
Reference Summary
Westfield: The sculpture was unveiled at London's 's Westfield Stratford City shopping centre.
James Corden: Benedict Cumberbatch is not the only famous face to be immortalised in chocolate. Jennifer Lawrence and James Corden have also heen turned into cakes.
Sophie Hunter: Actor, 38, is expecting his first child with wife Sophie Hunter.
CTRLsum
Jamie Dornam: Shoppers at London's 's Westfield Strafford City shopping centre looked baffled by the sculpture. Other famous faces to be immortalised edible art include Jamie Dornan and Jennifer Lawrence.
The sculpture was commissioned by UKTV to mark the screening of Sherlock. It follows in the footsteps of other A-listers such us Jamie Dornan and Jennifer Lawrence, who have been immortalised cakes. awrence's 's Hunger Games cake won an award show last year.
Hunger Games:
E.2
PAIRED ENTITY CONTROL
The entity control experiments in this paper only consider one entity as the control signal, here we show examples inputting paired entities. Specifically, we are interested in the output when one of the paired entities is important and the other is unimportant. Therefore, we sample 3 articles from CNNDM and for each article randomly select one important entity and one unimportant entity as paired keywords to show the model output. We repeat this sampling five times for each article to obtain five different summaries.
19

---

Under review as a conference paper at ICLR 2021
Table 16: Random Paired Entity Control Examples
A former U.S. Army enlistee who posted on Facebook about "the adrenaline rush" of dying in jihad was arrested Friday and charged with trying to detonate car homb at Fort Riley military base in Kansas, authorities said, A second mun, who allegedly knew about the bomb plot but didn't call authorities, was charged with failing to report felony. John T. Booker Jr. Topeka, an American citizen also known Mohammed Ahdullah Hussan, taken into custody near Manhattan, Kansus, the criminal complaint said, The "bomb" had actually been put together by two confidential informants with nonexplosive materials, the complaint said. Fort Riley's 's security was never breached und no people were in danger, U.S. Justice Department said in press releuse. Booker enlisted in the Army last year and was due to ship
that contained what he thought
bomb,
basic training April 7, 2014, said Anmy spokesman Wayne Hall. The criminal complaint suid the FBI questioned him March 24, 2014 about comments posted on Facebook. such as, "Getting ready to be killed in jihad is a HUGE adrenaline rush. I am so nervous. NOT because I'm scure to die but am eager to meet my lord." Booker waived his Miranda rights and told the agents he enlisted to commit an insider attack against American soldiers like Maj. Nidal Hassan had done at Fort Hood, Texas, the complaint suid, Hassun opened fire in a building in November 2009, killing 13 people and wounding more than 30. His enlistment was terminated March 24, 2014, at the request of Army Criminal Investigation Command, Hall said. Booker began communicating with confidential informant later in 2014, the complaint said, and often talked about his plans to engage in violent jihad support ISIS. He and the informant watched ISIS vidcos together, the complaint said, and Booker talked about how he wanted to go to Iraq and tumn his weapon on American soldiers when ordered to shoot the caemy. On March 9, Booker said he believed ISIS wanted him to commit a truck bombing in the United States and thought a good target would be nearby Fort Riley, a large Army base that's 's home to the Infantry Division, known as 'The Big Red One." Booker said "that detonating suicide homb is his No. I aspiration because he couldn't it be captured. all evidence would be destroyed and he would be guaranteed to hit his target." the criminal complaint said. He made a vidco with a Fort Riley airfield in the background and said ISIS was coming to kill American soldiers, both ahroad and in the United States, the complaint said. Booker aequired components The plan was for confidential informants to build a bomb and for Booker to drive to Fort Riley and detonate it, the complaint said. But the bomb was huilt with "inert" parts and would never explode, the complaint said, On Friday, the informants and Booker drove to what Booker thought was little-used utility gate near Fort Riley, the complaint said. While Booker was making final connections on the "homb," " the FBI arrested him, the complaint said. He was charged with one count of attempting to use weapon of mass destruction, one count of attempting to damage property by means of an explosive and one count of attempling to provide material support to the Islamic State of Iraq, a designated foreign terrorist organization. If convicted, he could face life in prison. Alexander Ft Hlair, 28, of Topeka was taken into custody Friday and charged with failing to report felony. The FBI said agents interviewed Blair after Booker's arrest. Bluir suid he shared some of Booker's views, knew of his plans to detonate vehicle bomb at Fort Riley and loaned him money rent storage space, according to the FBI's 's criminal complaint. He said he thought Booker would carry out his plan hut did not contact authorities, the complaint said. convicted, Blair faces maximum three years prison.
Atticle
bomb and rented a storage locker to store components, the complaint said.
Alexander Blair, 28, of Topeka accused of knowing about bomb plot but not contacting authorities. Fort Riley's security was never breached and the device was "inert" and not a threat, authorities say. John T.. Booker Jr., 20, of Topeka had acquired bomb parts and made propaganda video, the Justice Depurtment says.
Reference Summary
Fort Riley ISIS: John T. Booker Jr. is charged with trying to detonate car bomb at Fort Riley. A second man, who allegedly knew about the homb plot but didn't call authorities, is charged. Booker said he helieved ISIS wanted him to commit truck bombing in the United States.
John T. Booker Jr. FBE: John T. Booker Jr. enlisted in the Army last year and was due to ship out to busic truining April 7, 2014, The FBI questioned him March 24, 2014 about comments posted on Facebook.
John T. Booker Jr. Infantry Division: John T. Booker Jr. enlisted in the Army last year and was due to ship out to basic training April 7, 2014. He planned to detonate car bomb at Fort Riley, a large Army base that's 's home to the 1st Infantry Division.
CTRLsum
Justice Department Islamic State of Iraq: The Justice Department says Fort Riley's security was never breached and no people were in danger. John T. Booker Jr. is charged with attempting to use weapon of mass destruction. He is also charged with trying to provide material support to the Islamic State of Iraq.
Alexander Blair ISIS: John T. Booker Jr., 28. charged with trying to detonate car bomb Fort Ri- Icy. Alexander E. Blais, 28, is charged with failing to report felony. Booker said he wanted to commit insider attack against Americun soldiers in support of ISIS.
20

---

Under review as a conference paper at ICLR 2021
Table 17: Random Paired Entity Control Examples
Ander Herrera has caught the eye in recent weeks after cementing spot Manchester United's starting line-up but it appears he does not actually have a clear sight goal. Herreaa netted a brace against Aston Villa on Saturday alternoon without actually looking at Brad Guzan's 's goul as his eyes appeured to be shut when making contact with hall. fact, of Herrera's seven goals have at either the bull the opposition's net... Manchester United star Ander Herrera scores his side's opening goal against Aston Villa with his eyes shut. The Spunish midfielder appears to have his eyes closed as he strikes at Brad Guzan's is goal. Aston Villa and England midfielder Fabian Delph attempts to block Herrera's is left-footed shot. His eyes were wide open when he struck an impressive first-time shot against Yeovil in the third round of the FA Cup back January. However his double against Aston Villa and his gouls ugainst Queens Purk Rangers, Leicester. Preston and Swansea all came without Manchester United's 's summer signing having to make eye contact with the ball. Herrera appears to have a history of shooting with his eyes closed as the image of his goal Spuin's 's Under side back in 2009 shows. Herrera hits the back the net while representing Spain's 's Under 20 side against Tahiti. The former Athletic Bilbuo joined the Red Devils for E29million the summer due to his vision and creativity in midficld. Louis van Gaal heaped praise on the --ayrr-old's shooting ability after Saturday's 's Premier League encounter at Old Trufford, by stating: 'He has a very good kicking technique and should be more composed." Van Gaal said to MUTV. said him 'ou have to control the ball before you shoot'. 'I said that him again yesterday's training session, and then kissed him half-time because he had controlled the hall for the first time in his life and then he shot,' Herrera's 's eyes were also shut for Manchester United's ss third goal against Aston Villa on Saturday. His only Manchester United goal scored with eyes wide open came against Ycovil Town back in January Herrera scored in Manchester United's s 4-0 win over Queens Park Rangers without looking the hall. The former Athletic Bilbao man flicked the ball past Preston's 's players without having his eyes on the ball Herrera scored in Manchester United's 2-1 defeat by Swansea City at the Liberty Stadium in February. He also netted in Manchester United's is shock 5-3 defeat at the hands Premier League strigglers Leicester.
scored without him even having glimpse
Atticle
Ander Herrera has scored seven goals for Man United since joining in June. Herrera's 's eyes have been shut when striking the ball for six of his goals. His superb strike against Ycovil Town has been only goal with eyes open. Herrera netted brace in Manchester United's s 3-1 win over Aston Villa.
Reference Summary
Ander Herrera Van Gaal: Ander Herrera has scored six his seven goals without looking at the bull Louis van Gaal has praised the Spanish midfielder's shooting ability.
Yeovil Town Swansea City: Ander Herrra scored his side's opening goal against Aston Villa with his eyes shut. The 25-year-old has also scored without looking at either the ball or the opposition's 's net in games against QPR, Leicester City, Swansea and Ycovil Town.
Manchester United Queens Park Rangers: Ander Herrera appeared to have his eyes closed as he scored against Aston Villa on Saturday. The 25-year-old has scored six of his seven goals without looking at the ball. Herrera also netted with his eyes open Manchester United's 4-0 over Queens Park Rangers.
CTRLsum
Aston Villa MUTV: Ander Herera soored brace against Aston Villa on Saturday. But the Spanish midfielder appeared to have his eyes closed Louis van Gaal told MUTV that Herrera should be 'more composed"
Herrera Athletic Bilbao: Ander Herrera has scared six of his seven goals with his eyes closed. The former Athletic Bilbao man joined Manchester United for 229million in the summer.
21

---

Under review as a conference paper at ICLR 2021
Table 18: Random Paired Entity Control Examples
A bushand accused of trying to murder his cheating wife was ready to let her have with another man once month as long as she stayed with him, his trial heard yesterday, A close friend told how Joseph O' d'iiordan, 74, had confided in him about the extraordinary plan for his 47-year-old wife Mandy, O' R'Riordan, councillor and former nighteluh owner, stahhed her eight times in a jealous rage after finding out she had heen having affair with postmun. Extraordinary deul: Joseph O' O'Riordan stabbed his wife of ten years Amanda (left) with seven inch kitchen knife eight times yesterday Brighton Crown Court heard he was considering allowing her to have affairs. She suffered life-threatening injuries after being knifed in the torso, chest, arms and back. The jury was also shown dramatic footage of the moment police arrived at the couple's 's home to be greeted by a 'culm' O' Riordan opening the door. The revelation of his proposal for keeping his wife of len years cume from Alfred Hurris. He told how O' RRordan had confided five days before the attack that he believed she was having an affair. O' Riorrdan was 'choked up and emotional' when he said: 'I think Amanda is playing away. She's is getting her nails and hair done more regularly, she's 's been on diet and doesn't want sex.. Asking for suit: O' O'Riordan sent his wife this letter from his prison cell. The following day, added Mr Harris, the men met for pub lunch in O' O'Riordan's 's home village of Polegate, East Sussex. 'I saw Joe and he told me that Amanda had been seeing someone else- -a guy who drove van. Joe said he loved Amanda to bits and if she wanted to have sex with somcone else once month that would be okay as long as she stayed with him." Brighton Crown Court, Mr Harris also deseribed the couple 'loving and close". He was 'so shocked' learn that O'Riordan had attacked his wife at their flat a residential care home estate. The jury saw images of four police officers, one of whom was wearing a lapel camera, arriving shortly before 10pm last October 22 after racing to the scene. . PC Dave Catt said they drew their 'incapacitating' sprays fearing the knifeman would be still holding his weapon. They were greeted by Rioordan wearing blood-spatten spattered light blue shirt and holding cordless phone on which he had phoned for an ambulance. Mr Catt said O Riordan admitted: 'I found out that she was having an affair and lost it',. Mrs O' Riordan was moaning and lying on a bed, holding towel to her stomach with deep chest wound and serious wounds to her hand and hack. Paramedics arrived moments hospital. Jurors looked at two screens as images her hushand's arrest and subsequent detention police station were shown. Growing suspicion: Giving evidence yesterday Alfred Harris- a friend of the couple for more than six years- - told how O'Riordan had confided in him that he believed his wife was having an uffair. Yesterday, jurors at Brightom Crown Court (above) were shown dramatic foolage of the moment police arrived at the couple's home be greeted by 'calm' Mr O' O'Riordan opening the door. PC Stuart Kenway fold how, as O' Riordan had opened the door, he "appeared calm and composed and the situation was surreal' as he then said: 'She is in the bedroom- do you want the knife?" Officers were directed to 7in blude with a black hundle which was in the kitchen. Dr Stephen Drage, an intensive cure consultant with Brighton and Sussex University Hospitals,
a statement read to
Article
later and took
jury how seriously Mrs O' O'Riordan was hurt. is quite clear she was bleeding to death," he said. 'She underwent life-saving sungery which took six hours.. O' O'Riordun denies atlempted murder. The trial continues. Sorry are not currently accepting comments on this article.
Joseph R'oiirda,, 73, stabbed wife eight times after discowering affair. She was left with life- theethreatennng injuries to her torso, chest, and back. Yestedaay Brighton Crown Court heard about deal he was ready to offer her. He had told friend about the idea while in the pub just days before stabbing.
Reference Summary
Joseph O' O'Riordan Alfred Harris: Joseph O'Riordan, 74, is accused of stahbing wife Mandy, 47, eight times. Friend Alfred Harris told how he had told him about the extraoddinayy plan.
Brighton Crown Court Stephen Drage: Joseph O' O'Riordan, 74, accused of stabbing wife Mandy, 47, eight times. Brighton Crown Court heard he was considering allowing her to have affiirs, Dr Stephen Drage, an jury
intensive care consultant, told jury how she was 'clerrly hurt'
CTRLsum
Joseph O' O'Riordan Catt: Joseph O' O'Riordan, 74, is accused of stabbing wife Mundy, 47, eight times.
Dave said he 'lost it' when he found out about affair.
Stuart Kenway Joseph O' O'Riordan: Juseph O' O'Riordan, 74, is accused of stabbing wife Mandy, 47, cight times. PC Stuart Kenway told how he 'appeared calm and composed".
Brighton Crown Court Amanda: Joseph O' O'Riordan, 74, is accused of stabbing wife Amanda, 47, cight times. He denies attempted murder at Brighton Crown Court.
E.3
LENGTH CONTROL
For length control, we randomly sample 3 test articles from CNNDM and for each article we generate five different-length summaries.
22

---

Under review as a conference paper at ICLR 2021
Table 19: Random Length Control Examples. Control tokens are bolded.
Film director David Lynch has confirmed he will no longer direct the revival of "Twin Peaks" a cult 1990s television show that was set to return in 2016. The offheat TV series, created by Lynch and Mark Frost, featured quirky FBI agent who went to the Pacific Northwest town of Twin Peaks to investigute the mysterious murder of high school girl named Laura Palmer. The groundbreaking series considered one of the most influential shows television history. Lynch broke the news about his departure in series of tweels, saying that the show's s third season will continue without him. He said he felt the network was not offering enough money to produce the show "the way it needed to be done." Lynch also wrote that he had personally called the actors over the weekend to let them know would no longer be directing. Showtime Network, which will uir the nine-episode comeback, releused statement saying they were "saddened" by Lynch's decision. "We were saddened to read Duvid Lynch's 's statement today since we believed on the few remaining deal points," read the statement. "Showtime also loves the world of Twin Peaks and we continue to hold out hope that we can bring it back in all its glory with both of its extraordinary creators, David Lynch and Mark Frost. at its helm." Showtime announced they would produce third season in October last year. Actor Kyle MacLachlan, who played the obsece-obsesse FBI agent Dale Cooper in the original scries, had confirmed he would reprise the lead role for the new season.
Atticle
were working towurds solutions with David and his reps
Reference Summary
David Lynch says he won't he directing which involved dispute over money.
episodes of Twin Peaks, Showtime "saddened" over decision,
(Length bucket director David Lynch longer direct revival Twin Peaks return 2016 scason Showtime: Film director David Lynch will longer direct the revival of "Twin Peaks" Showtime announced they would produce third season in October last year. The show was set to return in 2016.
(Length bucket 1) director David Lynch longer direct revival Twin Peaks set return 2016 season Showtime Network saddened: Film director David Lynch will longer direct the revival of "Twin Peaks" " Showtime Network say they are "saddened" by the decision. The show was set to return for a third season in 2016.
(Length bucket 2) director David Lynch confirmed longer direct revival Twin Peaks set return 2016 I series season continue Showtime Network saddened: Film director David Lynch has confirmed will no longer direct the revival of "Twin Peaks" The series was set to return in 2016. Lynch broke the news about his departure in a series of tweets. He said the show's 's third seuson will continue without him. Showtime Network suid they were "saddened" by Lynch's 's decision.
CTRLsum
(Length bucket 3) director David Lynch confirmed longer direct revival Twin Peaks 1990s set re- turn 2016 series season continue Showtime Network saddened decision: SFilm director David Lynch has confirmed he will no longer direct the revival of "Twin Peaks" 1990s TV scries was set to return in 2016. Lynch broke the about his departure series of tweets. The show's third season continue without him. Showtime Network said they were "saddened" Lynch's decision.
(Length bucket 4) director David Lynch confirmed longer direct revival Twin Peaks cult 1990s set return 2016 series Mark Frost FBI agent murder tweets season continue Showtime Network statement saddened decision Kyle: Film director David Lynch has confirmed he will no longer direct the revival of "Twin Peaks" The cult 1990s TV series, created by Lynch and Mark Frost, was set to return in 2016. Lynch broke the news about his departure without him. Showtime Network, which will air nine-episode comeback, released a statement saying they were "saddened" by Lynch's decision. Actor Kyle Mael achlan, who played the coffee-ohsessed FBI agent Dale Cooper in the original series, had confirmed he would reprise the lead role for the new season.
series of tweets, saying that the show's 's third season will continue
23

---

Under review as a conference paper at ICLR 2021
Table 20: Random Length Control Examples. Control tokens are bolded.
Washington (CNN)An Iranian military ohservation aireraft flew within 50 yards of an armed U.S. Navy helicopter over the Persian Gulf this month, sparking concern that top Iranian commanders might not be in full control of local forces, CNN has leamned. The incident, which has not been publicly disclosed, troubled U.S. military officials hecause the unsafe maneuver could have triggered serious incident. It also surprised U.S. commanders because recent months Iranian forces have conducted exercises and operations in the region professional manner, one U.S. military official told CNN. "We think this might huve been locally ordered," the official said, The incident took place as the U.S. and other world powers meet with Iran in Switzerland to negotiate a deal limiting Tehran's 's nuclear program. At the same time, Iran has been active supporting proxies in several hotspots in the Persian Gulf and neighboring regions. The Navy MH-60R armed helicopter deck of the USS Carl Vinson routine patrol in international airspace, the official suid. An unarmed Iranian observation Y-12 aireraft upproached. The Iranian aireraft made two passes at the helicopter, corning within 50 yards, before helicopter moved off, according to the official. The official said the helicopter deliberately broke off and flew away in a 'predictable' manner so the Iranians could misinterpret any U.S. intentions. The Navy helicopter was in radio contact with the ship during the encounter, but there was no contact between the two aircraft and no shots were fired. The Navy crew took photos of the incident but the military is not releasing them. The U.S. administration is considering a potential demarche protest against Iran. the official said. CNN has reached out Iranian officials generally operates over the Gulf several times month. But after the recent incident, U.S. naval intelligence did not see it again for two weeks, leading to the conclusion that the incident may have been ordered by a local commander who was then reprimanded by higher-ups. The Pentagon has noted for the last several years that air conducted professionally.
flying from the
Article
has received response. This type Iranian observation aireraft
most encounters with the Iranian military by Iranian Revolutionary Guard Corps forces have been too aggressive against U.S. forces in the area. The U.S. military's 's concern has been that one of these incidents could escalate into military encounter. This incidntt "might have been buffoonery" the official said, hut there is always a risk from such actions. The incident comes as the Navy patrols the Gulf of Aden to watch Iranian ships the believes are trying to bring weapons to resupply the Houthi rebels in Yemen. The Navy would share such intelligence with Saudi Arabia, sccond U.S. official told CNN.
that some missions run
Reference Summary
Iranian plane came within 50 yards of U.S. Navy Sea Hawk copter. Navy copter was on patrol in international airspace. U.S. official think Iranian plane may have been under orders of local commander.
(Length bucket 0) Iranian military aireraft flew 50 yards armed U.S. Navy helicopter Gulf commanders: An Iranian military observation aircraft flew within 50 yards of an armed U.S. Navy helicopter over the Persian Gulf this month. The incident, which has not been publicly disclosed, has sparked concern that top Iranian commanders might be full control of local forces.
(Length bucket 1) Iranian military observation aircraft flew 50 yards armed U.S. Navy helicopter Gulf Iranian commanders forces: An Iranian military observation aircraft flew within 50 yards of an armed U.S. Navy helicopter over the Persian Gulf. The incident sparked concern that top Iranian commanders might not he in full control of local forces.
(Length backet 2) Tranian military observation aireraft flew 50 yards armed U.S. Navy helicopter Persian Gulf Tranian commanders control forces incident: An Tranian military observation aireraft flew within 50 yards of an armed U.S. Navy helicopter over the Persian Gulf. The inciden sparked concern that top Iranian commanders might not be in full control of local forces.
CTRLsum
(Length bucket 3) Iranian military observation aircraft flew 50 yards armed U.S. Navy helicopter Persian Gulf Tranian commanders control forces incident Iran protest: An Tranian military observation aireraft flew within 50 yards of an armed U.S Navy helicopier over the Persian Gulf. The incident sparked concern that top Iranian commanders might not be in full control of local forces. Iram has been active in supporting proxies in several hotspots the region. The U.S. administration is considering a potential demarche protest against Iran.
(Length bucket 4) Iranian military observation aircraft flew 50 yards armed U.S. Navy helicopter Persiun Gulf Iraniun communders control local forces incident officials official ordered Iran USS considering demarche protest Navy Aden: An Iranian military observation aireraft flew within 50 yards of armed U.S. Navy helicopter over the Persiun Gulf. The Navy MH-60R armed helicopter was flying from the deck of the USS Carl Vinson on a routine patrol in international airspace. The incident sparked concern that top Iranian commanders might not be in full control of local forces. "We think this might have been locally ordered," the official said. CNN has reached out Iranian officials but has not received a response. Incident comes as the Navy patrols the Gulf of Aden to watch for Iranian ships. The administration considering potential demurche protest against Iran.
24

---

Under review as a conference paper at ICLR 2021
Table 21: Random Length Control Examples. Control tokens are bolded.
It's sight that draws giggles and curious stares from tourists and other first-timers revellers carry gigantic phalluses through the streets of Japanese city. But for the residents of Kawasaki, who lug erotic shapes of all different sizes, this odd trudition is not a joke, Shinto Kanamara Matsuri started as small tradition has grown into a popular tourist attractonn, with participants praying a god of fertility, child hirth and protection from sexually transmitied infections. Participants carry a giguntic phallus through the streets of Kuwasaki, Jupan during the Shinto Kanamara Matsuri festival. The sight of three large phalluses being paraded through neighbourhoods in the city south of Tokyo draws giggles from tourists. Shinto Kanamara Matsuri. the Festival of the Steel Phallus, started as a small tradition but has grown inlo a popular a tourist attraction. Known the Festival of the Steel Phallus, is held every spring the phallus-shaped Kanayuma Shrine. Festivalgoers parade through the streets with three giant phalluses. while spectators lick lollies or snack on sausages or vegetables shaped us male and female genitalia Ruiny weather didn't I ruin the mood at this year's 's festival, which had a massive collection of foreigners, uccording to Japanese website RocketNews24. They watched as groups of locals carried three heavy phalluses modelled after mikoshi portable shrine, which commonly Shinto festivals, Residents of Kawasaki carry phalluses of all different sizes while participating in a tradition that began nearly 40 yeurs ago. Participants pray to a god who is suid to help with fertility. child birth and protection from sexually transmitted infections. Rainy weather didn't ruin the mood at this year's 's festival, which attracted thousands of spectators from Japan and abroad. Spectators can buy lollies, sausages and vegetables shaped male and female genitalia while warching the unusual parade. The three mikoshi carried through the streets for hours are the Kanamara Boat Mikoshi (tall and black). the Elizabeth Mikoshi (big and pink) and the Kanamara Big Mikoshi (squat and wooden), RocketNews24 reported. In addition to the parade there are performances and music, and the festival itself raises money and awareness for HIV research. The festival started nearly 40 years ago, and the Kanayama Shrine was once popular place for prostitutes pray for protection from sexually transmitted infections. Today, people visit the shrine to pray for everything from fertility and business prosperity to wedded bliss and good health. In addition to the parade there are performances and music, and the festival itself raises money and awareness for HIV research. The Kanayama Shrine was once a popular place prostitutes to pray for protection from sexually transmitted infections. Today, people visit the shrinc to pray for everything from fertility and business prosperity to wedded bliss and good health. Grrups of locals carried three heavy phalluses modelled after a mikoshi portable shrine, which is commonly used in Shinto festivals.
unusual festival where
in
Atticle
Shinto Kanamara Matsuri started small tradition but has grown into popular tourist attraction. Known as the Festival of the Steel Phallus, participants pray to a god of fertility and protection from infections. Rainy weather didn't 't ruin the mood at this year's festival, which attracted large crowd of holidaymakers.
Reference Summary
(Length bucket 0) Shinto Kanamara Matsuri started tradition god fertility child birth protection infections Steel: Shinto Kanamara Matsuri, the Festival of the Steel Phallus, started as a small tradition. Participants pray to a god of fertility, child birth and protection from sexually transmitted infections.
(Length bucker 1) Shinto Kanamara Matsuri started tradition praying god fertility child birth pro- tection sexually infections Festival Steel: Shinto Kanamara Matsuri, the Festival of the Stoel Phallus, started as small tradition. Participants pray to a god of fertility, child birth and protection from sexually transmitted infections.
(Length bucket 2) Shinto Kanamara Matsuri started tradition popular tourist praying god fertility child birth protection sexually infections Festival Steel Phallus: Shinto Kanamara Matsuri, the Festival of the Steel Phallus, started us small tradition but has grown into a popular a tourist attruction. Participants pray to god of fertility, child birth and protection from sexually transmitted infections.
CTRL
(Length bucket 3) Shinto Kunamara Matsuri started tradition popular tourist attraction praying god fertility child birth protection sexually transmitted infections Festival Steel Phallus: Shinto Kunamara Matsuri, the Festival of the Steel Phallus, started as small tradition but has grown into popular a tourist attraction. Purticipants pray to god of fertility, child birth and protection from sexually transmitted infections.
(Length bucket 4) phalluses Shinto Kanamara Matsuri started small tradition grown popular tourist attraction praying god fertility child birth protection sexually transmitted infections Kawasaki Japan Matsuri festival Tokyo Festival Steel Phallus I 40: Shinto Kanamara Matsuri started as a small tradition 40 years ago. has grown into a popular tourist attraction in Kawasaki. Japan. The Festival of the Steel Phallus held every spring the Kanayama Shrine. Participants carry three phalluses through neighbourhoods south of Tokyo. They are praying to a god of fertility, child birth and protection from sexually transmitted infections. Thousands of tourists attended this year's 's festival, which raised money for HIV reseurch.
E.4 CONTRIBUTION SUMMARIZATION ON SCIENTIFIC PAPERS
Here we show three random examples from the arXiv test set. Note that this is the test set from (Cohan) et al. 2018) instead of the contribution test data collected by us, because we want to show the differ- ence between reference summaries (i.e. the paper abstract) in existing standard paper summarization dataset and our output contribution summaries. We truncate the source articles since they are too long to display.
25

---

Under review as a conference paper at ICLR 2021
Random Contribution Summarization Examples. Control tokens are bolded. "I"" denote that the Table 22:
tokens are used as both keywords and prompts.
synchronization of neural activity appears different parts the mammalian cerebral cortex @ @xiite, , and underlies different neural processes in both normal and anomalous brain functions @xcile. it has been suggested that synchronization plays a vital role in information processing in the brain. e.g. processing information from different sensory systems to form coherent and unified perception of the external world @ xcite on the other hand, , synchromization has been detected in pathological conditions such as parkinson s disease @ xcite . and epileptic seizures have long been considered resulting from excessive synchromized brain activity xcite, though some recent studies suggest that this picture may he an over - simplification @ xciite therefore understanding the mechanisms of synchronization may be critical step in elucidating how neural systems work @ xcite. it has stimulated great deal of theoreticul and numerical works, such as the studies on the effects of the topological properties of underlying networks @exile and the dynamical proerties of synaptic coupling @ xxite. was recently shown that the response time of synaptic couphings influences the stability of synchromized uscillation in the nonlocully coupled hodgkin - huxley ( (hh) ) equations @xcite if the response time of synaptic coupling is slower, synchronized activity of the systems is instable for excitatory coupling. however, the underlying dynumical mechunism of the influence is not clear
experimental studies @ xciie. has been suggested that the generation of prolonged epileptiform neuronal synchronizution is favored by lower efficacy of synaptic transmission. the numerical studies @xcite in a detailed computational model revealed that scizure- like activity occurs when the excitatory synupses are weakened, and the were confirmed experimentally neocortical slices. according the common accepted assumption that synchronization of neuronal activity underlies scizures. the dynamical mechanism of synchronization may be useful for understanding the way the biological neural system works. in this work, we numerically investigated the dynamical mechanism underlying the influence of synaptic efficacy on firing synchronization in hh neuron networks. to do this, we first studied the dynamics of the response of hh neuron exeitatory synaptic current. when the efficacy of synapse low namely. strength weak and duration is short, the limit cycle is stable to the perturbation of the synaptic current. when synaptic efficacy is high. synaptic current can induce the transition of the neurons from limit eycle to fixed point or transient state the transition is determined by dynamics of neuron s ionic channel. the decrcase of synaptic current depresses the feedback sodium ionic current which is responsible for the initiation of the spike. for simplicity we will refer the transitions spike death. in neuronal networks, synaptic input of neuron is the accumulation of the currents received from all presynaptic ncurons. when the coherence of firing time of neurons is enhanced by the excitatory interaction, the synaptic input of neurons transforms from the fluctuating waveform into the pulse shape like signal produced by one synapse. if synaptic efficacy is high, the input signal rhythm of neurons and prevents neurons from firing spikes synchronously. in contrast, for synapses of lower efficacy. the duration of synaptic current is too short to induce spike death of neurons. additionally the firing sy nccromizzaionn is different from synchromous activity oscillators for the existence state. the paper is organized as follows. the hh neuron model and the synaptic coupling introduced in sec, the response of hh neuron to synaptic current was investigated in sec. the influence of the dynamies of neurons on firing synchrony was shown in sec.
mouse
Article
induce spike death of the neuron. then spike death disorders the adjustment of the
transitions neuron
investigated the influence of efficacy of synaptic interaction on firing synchronization in excitatory neuronal networks. we found spike death phenomena. namely. the state of neurons transits from limit cycle to fixed point or transient state the phenomena occur under the perturbation of excitatory synaptic interaction that has a high efficacy. . we showed that the decrease of synaptic current results in spike death through depressing the feedback of sodium ionic current. in the networks with spike death property the degree of synchronization is lower and unsensitive to the heterogeneity of neurons. the mechanism of the influence is that the trunsition of neuron state disrupts the adjustment of the rhythm of neuron oscillation and prevents further increase of firing sy synchronization.
Reference Summary
(the main contributions of this paper are : (1 }]: we investigated the dynamical mechanism underlying the influence of synaptic efficacy on firing synchrony in hodgkin - huxley neuron networks: ( we found that the dynamics of synaptic current plays an important role in determining the stability firing synchronization.
CTRL.um
26

---

Under review as a conference paper at ICLR 2021
Table 23:
Random Contribution Summarization Examples. Control tokens are bolded. "I"" denote that the tokens are used as both keywords and prompts.
the understanding of surface reactions and the characterization of materials it desirahle measure local forces close to sample surface.. the most common method to measure these surface forces atomic force microscopy ( afm) @ xeite. historically. the first force measurements were static measurements for which the force is presented as a scalar function of the static tip - sample separation, the so - called force curve xcite. this representation sufficient for conservative forces but the total tip- - surface force may also contain contributions from dissipative forces. since dissipative forces depend on probe velocity and pust trajectory dynamic force spectiescopyy methods are required for their measurement. moreover, the visualization of dissipative forces as a function of position is valid only for a specific probe trujectory and simple force curves can not capture the full churacter of the interaction. despite the development of several dynamic methods @xcite surface forces are still usually treated as functions of the probe position only and represented by simple force curves. here, comprehensive frumework for the represenlation and analysis of complex surface forces as they are measured by dynamic afin. we concentrate on the most common modes of dynamic afm: : amplitde - modulated afm am- afm and frequency - modulated afm fm afmn) ). which can be considered as narrow frequency band methods @ xcite... we explore the fundamental limit of force reconstruction with narrow bund dynumic afm at fixed probe height and show how minimal assumptions allow for quantitative reconstruction of the tip- surface interaction. at the heart of the afm apparatus is a micro- cantilever with a sharp tip. the cantilever is firmly clamped at one end and the tip located at the other end which can move freely. it is assumed that surface forces only act on the tip whereas the of the cantilever does not experience significant surface forces oscillatory motion.. thus the
present
Atticle
in dynamic afm an additional external drive force is applied to maintain dy dynamics are governed by the force between tip and surface the extemal drive foree and the properties of the cantilever beam... since the cantilever is a three dimensional continuum object its motion is usually described by general. these modes
the amplitudes of different oscillation eigenmodes. all directions in space. however, the cantilever is positioned such that the softest flexural modes bend the beam in a plane orthogonal to the surface plane. we restrict ourselves to the case where only these flexural modes are excited by the drive forco. due this experimental configuration the cantilever is much more susceptible to the component the tip- surface foree which is orthogonal to the surface plane . this component of the force is typically the most dominant component and the influence of lateral force components considered negligible.. in this case the cantilever acts as a mechanical projector which reacts only to one component of a three dimensional force vector field, , the deflection @ 6xmath0 of cantilever of length @ xmathh orthogonal to surface descrihed by one dimensional euler hermoulli equation @ xcite @ xmath2 where xmath3 is the young modulus. @ xxmuth4 is the second moment of area, @ xmatth5 is the mass per unit length of the cantilever, @xmath6 is the position coordinate along the cantilever beam and @xmath? is the time variable the force term @ @xmat88 includes the surface forces acting as a point like load at position @ xmath9, the external drive force and the hydrudymamic damping due to the surrounding medium xcite.
cause cantilever to bend
in atomic force microscopy ( (afm) ) tip surface interactions are usually considered as functions of the tip position only. so - called force curves however, tip surface interactions often depend on the tip velocity and the past tip trajectory here, introduce compact and general description of these interactions appropriate dynamic afm where the measurement of force is restricted to narrow frequency band. represent the tip- surface interaction in terms of a force disk in the phase space of position and velucity determination of the amplitude dependence of tip surface forces at fixed static probe height allows for a comprehensive treatment of conservative and dissipative interactions. we illuminate the fundamental limitations of force reconstruction with narrow band dynamic afm and show how the amplitude dependence of the fourier component of the force at the tip oscillation frequency. gives qualitative insight into the detailed nature of the tip surface interaction. with minimal assumptions this amplitude dependence force spectroscopy allows for a quantitative reconstruction of the effective conservative tip - surface demonstrate this reconstruction simulated intermodulation afm data. key _: atomic force micrsscopy. measurement of force, mechanical resonators, mems/ / nems, dissipation. internodulation
Reference Summary
as well as position dependent damping factor
[the main contributions of this paper are : ( (1) )}: a comprehensive framework for the representation and unalysis of complex surface forces they measured by dynamic atomic force microscopy aff (2) study of the fundamental limit of force reconstruction with narrow band dynamic afm at fixed prohe height and show how minimal ussumptions allow for a quantitative reconstruction of the tip surface interaction. .
CTRLsum
27

---

Under review as a conference paper at ICLR 2021
Table 24:
Random Contribution Summarization Examples. Control tokens are bolded. "I"" denote that the tokens are used as both keywords and prompts.
in this paper we discuss the mathematical aspects problems originating the solution of nonlinear systems of hyperbolic partial differential equations.. these equations describe large variety of physicul phenomena, mhh) ), shallow water equations, clasticity equations. etc...
such as gasdynamics magnctohydrodynaae being nonlinear, , these systems usually require numerical methods for their solution. . presence discontinuous solutions motivates necessity the development reliable numerical methods based on the fundamental mathematical properties of hyperbolic systems although such methods rather well developed for the euler gasdynamicc equations in the conservation law form, their extension to more complicated hyperholic systems is not straightforwat it requires a mathematical justification of the solution uniqueness. a formulation of the selection principles for relevant solutions, , and finally... an investigation their physicul validity. . most of high resolution methods for gasdynamic equations use the exact some of the approximate self similar riemann problem solutions to determine fluxes through the computational cell surfaces. similar methods are expected to be developed for various types of hyperbolic systems. in this case we must construct the elementary self' - similar solution using only admissible discontinuities entropy consistent evolutionary etcc ). basically the choice of the solution must be made on the basis of the structure of the solution of the extended problem @ xcite all mentioned above makes very important the study of discontinuous solutions behavior under vunishing viscosity and dispersion to create a proper background for the development of high - resolution numerical methods for hyperbolic systems more complicated than the euler equations of gasdynamics we discuss several anulytical and numerical solutions in the mentioned fields which illustrate the complexity of the selection problem and outline the methods of its solution. tvd upwind and symmetric differencing schemes have recently become efficient tool for solving complex multi- shocked gasdynamic flows. . this is due to their robustness for strong shock wave culculations... the extension of these schemes to the equations the ideal seggnetohyrrdyyamaee simple. . first. the exact solution xcite the midd riemann problem calculations... second, several different approximate solvers @xcite. @ xcite. @xcite, @xcite, @xcite. @ xeiiie and @ xcite applied to mhd equations are now at the stage of investigation and comparison. this investigation requires i) determination of a proper slope limiting method in the parameter interpolation procedure necessary to obtain nonoscillatory schemes of the order of accuracy higher than one ; ii development of efficient entropy correction method necessary exclude rarefaction shocks; and finally. iii solution of the problem of excluding the origin of nonevolationary solutions in idcal mhd calculations . the system of governing equations for a mhd flow of an ideal, infinitely conducting, perfect plasma in the cartesian coordinate system @ @xmmthh) @ kmath1 @ xmmath2 with the use of the conventional notations reads ( (one fluid approximation ): @ xmaath3 where @ xmmth4 is the vector of conservative variables and @ @xmath5, @ xxmath6, and @ xmath? are the flux vectors. we introduced here the source term @xmath8 in the form @ #xmath9 this form of the system can be used to satisfy the divergence free condition by convecting away the magnetic charge from the computational region @ xcite. otherwise, any other well known method can used to eliminate the magnetic charge to determine a numerical flux @ (xxmat10 10 normal to the computational cell houndary ( @ @xmathl I is a unit outward vector normaal to the cell surface ) one can use the formulas based on the solution of the linearized problem @ xmath 12. ] I here @ xmath13 and @ xmmuhh14 14 are the mutrices formed by the right and by the left eigenvectors, respectively . of the frozen jacobian matrix @ @xmath15 the matrix @xmath16 the frozen jacobian matrix eigenvalue moduli. the superseripts @ xxmat117 7 and @ @xmath18 denote the values the right- and at the left hand side of the cell boundury.
Attiele
is multivariant to be used regular
diagonal matrix consisting of
number of physical phenomena are deseribed hy nonlinear hyperbolic equations. presence of discontinuous solutions motivates the necessity of development of reliable numerical methods based on the lunddamentl mathematical properties of hyperbolic systems. construction of such methods for systems more complicated than the culer gas dynamic equations requires the investigation of existence and uniqueness of the self similar solutions to used in development discontinuity- - capturing high resolution numerical methods. this frequently necessitates the study of the behuvior of discontinutiis under vanishing viscosity and dispersion di discuss these prohlems in the application to the magnetohydrrolymami equations, nonlinear waves in elastic media, , and electromagnetic wave propagation in magnetics.
Reference Summary
[the main contributions of this paper are: (1 }]: the mathematicul aspects of the problems originating in the solution of nonlineat systems of hyperholic partial differential equations ; 2 the study of discontinuous solutions behavior under vanishing viscosity and dispersion to create proper grongr for the development of high- resolution numerical methods for hyperbola systems more complicated than the culer equations of gasdynamics and (3) solution of the problem of excluding the origin nonevolutionary solutions in ideal magnetuhydnudymumiss calculations.
CTRLsum
E.5 INVENTION PURPOSE SUMMARIZATION ON PATENT FILINGS
Here we show three random examples from the BIGPATENT test set. Note that this is the test set from origial BIGPATENT, because we want to show the difference between reference summaries in existing standard dataset and our output purpose summaries. We truncate the source articles since they are too long to display.
28

---

Under review as a conference paper at ICLR 2021
Table 25:
Random Invention Purpose Summarization Examples. Control tokens bolded. "I"" denote that the tokens are used as both keywords and prompts.
referring the drawings and in particular to figl therein illustrated a prior art surgical support mesh mesh 10 may be manufactured from monofilament multifilamet yams. prior mesh 10, shown includes multifilament borizontally extending yarns 12 and multiflament vertically extending yarns 14 woven together to form a support trellis. the use multiflament yarns, such as yarns 12 and 14, provides a mesh having greater pliability and suppleness than the use of monofilament yarns. these churacteristics result from both the smaller diameter of the individual filaments and the interstitial spaces voids that are located between such filaments.. in particular the flexibility of a filament or fiber) ) generally increases as its diameter decreuses because the solid cross sectional urea of the filaments of a multifilamet yarn is less thum the cross sectional area of monofilament yam of equivalent diameter, the multifilament yurn will have greater degree of flexibility and pliability than that of the monofilament yam. as shown in figl each of multifllment yarns 12 and 14 is composed of plurality of filaments 16 that are interminngeed or bundled together to form the yam. interstitial spaces 18, which are pockets of air, are formed between adjacent filaments of the yamn although these voids contribute to the softness and pliability of the formed mesh they also provide a natural breeding ground for bucteria other infectious material. sungical mesh is of course. thoroughly sterilized prior to implantation. nevertheless. surgeons typically prefer the use of monotilament designed mesh to minimize any risk of infection.. as a result, the advantages associated with multifilmmnnt - designed mesh (i. c.,, softness and pliability which result in betler ussimilation of the mesh into the body are typically sacrificed it has been discovered herein that surgical support mesh having both the softness and pliability of a multiflament- designed mesh and the infection resistance of monofilament- designed mesh may be produced. particularly it has been discovered that a support trellis formed of multifilament yam wherein the interstitial voids located between adjacent filaments are enclosed within an infection- - impervious matrix exhibits the desired resistance to harboring of infectious matter without significant loss ficxibility. particularly the matrix. which completely encloses the interstitial voids between the filaments of the yamn provides an effective barrier the passage of infectious matter between the interior and exterior of the yarn. accordingly any voids remaining in the yarn after encapsulation of such yarn are enclosed and thereby scaled) within the resultant matrix. first embodiment the present invention shown fig?. particularly . this embodiment includes a support trellis 20 formed of multifilamen yams 22 and 24 which overlap cross forming of the trellis. , such trellis is encapsulated within a matrix 26. which is preferably flexible material that continuously surrounds the exterior of the yarns therehy enclosing interstitial voids 27 located between filaments 28 see fig2 ). one embodiment, , the matrix formed from polymeric resin as shown fig2 a, the resin can be upplied the yurn in such particularly. the penetration of the resin can be controlled through the application procedure e 8 quantity of resin applied and / or encapsulating time. in such an embodiment, the interstitial spaces are enclosed ( rrather than filled ) within
Atticle
junctions 25. subsequent to
not allow the resin to substantallly penetrate into the yumn..
continuous matrix. however, is contemplated that the resin can be allowed to penetrate into the yarn. thereby substantially filling the void space located therein. in another embodiment of the present invention individual yarns 29, as shown in fig. are encapsulated within matrix 30 prior to forming of the .
support trellis. fig3 shows a compressed yarn 29 which provides a trellis having reduced thickness us u result of the encapsulation. interstitial voids 32 remaining in the yarn are enclosed ( (nd thereby scaled) within the matrix.
soft and pliahle surgical support mesh exhibiting increased resistance inhabitation of infectious matter. the mesh includes support trellis formed of multifitament yams wherein the interstitial voids located between the filaments of said yarns are enclosed within an infection- impervious matrix. the meshes may be designed to be extremely thin yet retain the requisite strength for repairing soft tissue, which allows for folded for delivery.
Reference Summary
profile when
[the purpose of the present invention is]: to provide surgical mesh that is resistant to the growth of bacteria and other infectious matter. this accomplished by encapsulating interstitial spaces located hetween the filuments of the yurn within matrix.
CTRLsum
29

---

Under review as a conference paper at ICLR 2021
Table 26: Random Invention Purpose Summarization Examples. Control tokens bolded. "I"" denote that the tokens are used as both keywords and prompts.
figl shows a multicolor weh fed rotary printing press accordance with invention. the press includes four arrangements 2 a. 2 2 and 2 d for printing single color multicolor image on the webs a 4 b, 4 c.cand and 4 d. the webs a, 4 b. 4 c and 4 d travel in substantially lincar direction through cach of the towers 2a 2a-2 2., . for example, , the weh can travel along substantially vertical path, , as shown in fig1 alternatively. as those skilled in art will appreciate the web path can be a substantially horizontal direction or in a substantially lineur path any desired ungle relative the vertical direction shown. the towers 2 a 2 d each include four printing units c. 6 6 y and 6 b for respectively printing an image in cyan, magenta yellow and black on buth sides of each web a- 4 d. other colors besides cyun, magenta, yellow and black can be used. the webs can be for example. , between 1200 and 1600 millimeters wide each of the printing units 6 6 m. y and 6 b in tower can he moved along respective weh 4 by lifting and positioning system 8 shown in fig2.. the listing and pusitioning system 8 includes spindle drive 10, which has fixed spindle 12 spunning range 14 over which the printing units 6 . 6 m 6 und b can be moved each of the printing units c. m. 6 y and 6 includes a ball screw 16 which rotatably supported in housing 18 the ball screw 16 can be rolated by motor 20 shown in fig2. fig2 shows one set of spindle drive 10, fixed spindle 12. ball screws 16. and motors 20. but preferably each tower 2 is provided with several sets one set for each corner of the print unit housing 18 the motors 20 are controlled by a motor control unit 22. which receives communds from a remote control 24. by pressing button on remote control 24, an operator cun control the rotation of the motors 20 and thereby the movement direction and position of each printing unit 6 y 6 m and 6 tower 2. rail systems not shown ) fixed to a side frame of each tower 2 can also be used to precisely guide movements of the printing units in the tower as shown in figl I and 2, the position of cach of the printing units 6 along the webs 4 and fixed spindles 12 can be controlled by the operator to allow access to a desired part of printing unit for example. . in figl position where a printing plate of the print unit b can be most casily accessed after the printing plate has been accessed, , the operator can move the print unit 6 b into a different position so that inker units in an upper part the printing unit 6 b can be casily accessed two or more printing units in tower 2 can also be moved as group. for example. shown in figl he can simply move the two printing units 6 y. 6 upwardly together until the top surface 28 of the printing unit y contacts the bottom surface 30 of the printing unit 6 thereafter, the operator can move the group of printing units 6 b. 6 and m upwards into the position shown tower 2 of figl where the plate eylinder of the printing unit
Atticle
operator has moved the print unit
tower 2 into
operator wants access the plate cylinder
printing unit m tower b
can easily accessed although figl and show four printing units for each lower 2. different numbers of printing units can be used for each lower. figl shows that the lower three printing units 6 of the tower 2 b are beneath an operating floor 26. preferably at least two of the printing units can be lowered beneath the operating floor 26, and printing press in accordance with the invention can be configured so that all of the printing units in a lower can lowered beneath an operating floor the operating floor can be a floor of print shop, can be elevated platform.
the present invention directed to a multicolor web fed mtaay printing press having printing units that can be moved along a linear section web by a positioning mechanism to allow casy access each of the printing units the easy access the printing units significantly reduces maintenance costs and downtime of the press. in addition, the invention provides a printing press that has a reduced overall size and allows the printing units to be arranged in a nested formation during printing operations this nested formation reduces fanout and paper waste. the linear weh section can be oriented vertically or horizontally.
Reference Summary
[the purpose of the present invention is]: to provide web fed rotary printing press that allows an operator to access desired part of a printing unit more casily than in the past.
CTRL.um
30

---

Under review as a conference paper at ICLR 2021
Table 27:
Random Invention Purpose Summarization Examples. Control tokens are bolded. "I"" denote that the tokens are used as both key woywords and prompts.
referring now the drawings and more particularly figl 3 fluid system 10 illustrated and includes
variable displacement axial piston pump 12 that receives fluid from tank 14 via conduit 16 and delivers pressurized fluid via a supply conduit 18 to a fluid control valve 20 and selectively through work conduits 22 24 to a fluid actuator 26. in the subject arrangement the variahle displacement axial piston pump 12 is a unidirectional pump that rotates
counterclos lockwiss direction as driven by a power inpui shaft 27. the fluid system 10 also includes first and second pressure sensors 28, 30 respectively connected to the tank conduit 16 and the supply conduit 18. the pressure sensors 28, 30 are operative to sense the pressure in the respective lines and deliver an electrical signul to controller 32 through electricul lines 34, 36 position sensor 40 is mounted on the variable displacement axial piston pump 12 and operative sense the displacement of the pump and deliver signal representative thereof the controller 32 via electrical line 42, various other components could be used in the subject fluid system 10 without depurting from the essence of the subject invention. for example, several control valves 20 and associated fluid actuators 26 could be used, likewise. , other sensors of various types and styles could be used the variable displacement axial piston pump 12 includes housing 44 having head portion 46 and a body portion 48 the head portion 46 defines un inlet port passage 50 that is connected to the conduit 16 and an outlet port passage 52 that connected to the supply conduit 18. in the subject arrangement a port plate 54 is disposed between the head portion 46 and the body portion 48 the construction of the porting within the port plate 54 is more clearly illustrated in fig3 and will be discussed more fully below. it is recognized that the potting illustrated fig3 could be made within the head portion 46 without departing from the essence of the subject invention. a rotating group 56 is disposed within the body portion 48 and includes barrel 58 having a plurality of cylinder bores 59 defined therein spaced from one another around an axis rotation the axis of rotation 60. a plurality of piston assemblies 62 are operatively associated with the barrel 58 and cach one of the plurality of piston assemblies 62 includes piston 64 slidcably disposed in the respective ones of the plurality of cylinder hores 59 each one of the plurality of piston assemblies 62 also has shoe 66 pivotably attached to one end of each piston 64 in a conventional manner. the barrel 58 has an end surface 68 that in provide communication between the cylinder bores 58 and
Atticle
the barrel 58. each of the cylinder bores 59 is oriented within the barrel 58 parallel with
mating, , scaling contact with the port plate the respective inlet and outlet post passages 50. 52 of the head portion 46. closed chamber 70 is defined in each cylinder bore 59 of the barrel 58 between the end of the piston G4 and the end surface 68 thereof. referring to fig3 the porting between the harrel 58 and inlet and outlet port passages 50 52 of the head portion 46 is more clearly illustrated
explanation purposes only. , the 270 degrre position illustrated in fig3 relates to position on the right side of the drawing figl and the 0 deggree position illustrated in fig3 relates to position on the ri right side of the drawing of fig?. an arcuate slot 72 is defined in the port plate 54 and provides communication between the plurality of closed chamhers 70 and the inlet port passage 50 plurality of slots are defined in the port plate circumferentially spaced from urcuate slot 72 and provides communication between the plurality of closed chambers 70 and the outlet port passage 52.
a variable di displacement axial piston pump is typically used to receive fluid from a tank and supply pressurized fluid through control valve move an actuator the present variable displacement axial piston pump has swashplate arrangement that is capable of being angled in two different directions to control the pressure transitions between the low pressure inlet port passage and the higher pressure outlet port passage as cylinder bores in barrel of a rotating group rotate through trapped volume regions situated between inlet and outlet port passages the axial piston pump- movement of the swashplate arrangemen two different directions provides smooth pressure transitions and increases the operating efficiency of the variable displacement axial piston pump
Reference Summary
[the purpose of the present invention is] provide variahle displacement axial piston pump that is capable of delivering a variable amount of pressurized fluid response change in the displacement of the pump
CTRLsum
E.6 QUESTION-GUIDED SUMMARIZATION
We randomly sample 3 article from NewsQA and show five questions and answers from CTRLsum for each article. We also show the gold answers to these questions.
31

---

Under review as a conference paper at ICLR 2021
Table 28:
Random Examples Question- guidedd summarization. Control tokens are bolded. "I"" denote that the tokens are used as both keywords and prompts.
TEHRAN, Iran (CNN) Iran's parliament speaker has criticized U.S. President-elect Barack Obama for saying that Iran's 's development of nuclear weapon unacceptable. Iranian President Mahmoud Ahmadinejad has outlined where he thinks U.S. policy needs to change. Ali Larijani said Suturday that Obama should upply his campaign message of change to U.S. dealings with Iran. "Ohama must know that the change that he talks about not simply superficial changing colors or tuctics," Larijani said cumments carried by the semiiofficial change in strategy, not the repetition of objections Irun's 's nucleur
Mehr News Agency. "What expected program, which will be taking step in the wrong direction." " In his first post-clection news conference Friday afternoon, Obama reitcrated that he believes a nuclcar-ar armed Iran would be "unacceptable." He also said he would help mount an international effort to prevent it from happening. Larijani said that U.S. behavior toward Iran "will change simply" but that Obama's 's election showed internal conditions in the United States have shifted. He added that Iran does not mind if the United States provides other Persian Gulf countries with nuclear technology, but "you should know that you cannot prevent the Islamic Republic [from reaching its goals in the nuclear field]," " according to the news agency. Ohama cautioned Friday that it had only been few days since election and he was not office. "Obviously, how we approach and not something that should simply do knee-jerk fashion. think we've said. "But have to reiterate once again that we only have one president al a time. And I want to be very careful that we are sending the right signuls to the world as whole that am not the president, and won't I be until Junuary 20th." Larijani was speaking two days after Iranian President Mahmoud Ahmadinejad congratulated Obama, the first time Iranian leader has offered such wishes U.S. president-clect since the 1979 Islamic Revolution. One unalyst said the welcome was gesture from the hard-line president that he is open to more conciliatory relationship with the United States. Ahmadinejad suid Tehran 'welcomes basic and fair changes in U.S. policies and conducts." according to the rate-run Islamic Republic News Ageney on Thursday. Relations between the United States and Iran have historically been chilly and have been further strained in recent years over Iran's 's nuclear program. Tehran insists that the program exists for peaceful purposes, but the United States and other Western nations are concerned by Iran's 's's refusal to halt uranium enrichment activitics. CNN's Shirzad Bozorgmehr contributed to this report.
Article
with a country like is to think it through," Obama
[Q: Iran criticizes who? A:]: U.S. President-clect Barack Obama. Gold Answer: U.S. President-elect Barack Obama
What are US and Iran relations tensioned about? A:]: Iran's nuclear program. Gold Answer: nuclear program.
[Q: Who said Obama should apply campaign message? A:]: Ali Larijani, Irun's parliament speaker. Gold Answer: Larijani
CTRLsum
[Q: What should Obama apply according to speaker? A:]: Change in strategy, not the repetition of objections.
Gold Answer: his campaign message of change
[Q: What kind of weapons are being discussed? A:]: Nuclear weapons.
Gold Answer: nuclear
32

---

Under review as a conference paper at ICLR 2021
Table 29:
Random Examples Question- guidedd summarization. Control tokens are bolded. "I"" denote that the tokens are used as both keywords and prompts.
Editor's note: The staff at CNN.com has recently been intrigued by journalism of VICE, independent media compuny and website based Brouklyn. New York. VBS.TV is Vice's broadband lelevision network. The reports. which are produced solely by VICE, reflect transpurent approuch to journulism, where viewers are taken along on every step of the reporting process. We believe this unique reporting approach is worthy of sharing with our CNN.com readers. Brooklyn, New York (VBS.TV) Whenever someone at our office stock response "Beats digging ditches." While the express intention
bitches about being overworked, of the statement is usually not-so-supportive. we think it's healthy reminder that at the end of the day. we are all basically professional e- e-mailers and should be thankful for such. The wildland firefighters who work for Grayback Forestry in Medford, Oregon, have no such motivational adages because their job is actually digging ditches. Around active forest fires. On the sides of mountains. You can't even bitch these guys for having cushy govermment pensions to fall back on when they get older, because they'n re all private-sector contractors. Which mcans if they aren't 't out fighting forest fires or doing preventative forestry on unburned woods (hasically extreme landscaping), they are losing money. They are the hardest working men the tree business. Souther Oregon majority of which occurred in the seemingly endless of trees running across its bottom from the Cascades the Pacific Coast. Humidity is next to nonexistent, which is extremely pleasant, but means that even an errant spark from chainsaw or the proverbial cigurette butl out the cur window can set the entire region ablaze. Fllinng into the Rogue Valley, there is evidence of past wild fires is everywhere: From the miles-long scar of the 2002 "Biscuit" fire stretching past the horizon. to the smaller pockets of charred trees crowding the edge of towns to the blue Wilderness- Dallerress-Iirefihttteee irrefiggheerr-ibbbbeee firefighter shrines at a locul bur. See more of Oregon Fire Lines at VBS.TV TV There ure very few places firemen aren't 't revered as local heroes, but the wildemess firefighters of Oregon go past people simply risking their lives help others into crazy superhcro realm where their work regularly prevents entire settlements from being destroyed. They'r re literally the guardians of their communities. Incidentally, their work is also very beneficial for the forest. We spent few days following a crew of Grayback forest-firefighes most people would consider a cliff, to chop down underbrush in preparation for a controlled burn. This what they like call "prject work" work we have tried do, Unless you consider 12- 12-hour-plus shifts of backhreaking labor, virtually zero outside recognition, and occasional accusations of being shills for the timber industry rewards. Which we do not. This piece was originally produced in August 2010.
summer is tinderbox. Last year the state recorded some wild fires, the
Article
bumper stickers flying pusi un the 1-5 to the elaborate wildfire and
walk up the sides of what
light stuff they hetween fires. The work the hardest and least rewarding
Q: What does the VBS call it? A:]: Project work. Gold Answer: "B "Beats digging ditches." "
[Q: How long are the shifts? Az]: 12-hour shifts. Gold Answer: 12-hour-pius
[Q: What does the 12 hr shift involved? A:]: Project work.
CTRLsum
Gold Auswer: backbreaking labor, virtually zero outside recognition, and occasional accusations of being shills for the timber industry
[Q: Which group followed firefighters? A:]: The wildland firefighters of Oregon. Gold Answer: VRS.
[Q: What is the VBS following? A:] Wildland firefighters who work around active forest fires.
Gold Answer: a crew of Grayback forest firffghere
33

---

Under review as a conference paper at ICLR 2021
Table 30:
Random Examples Question- guided summarization. Control tokens are bolded. "I"" denote that the tokens are used as both keywords and prompts.
WASHINGTOON (CNN) The nation's largest publicly owned utility company may be vulnerable cyber atlacks, according by the Tennessee Valley Authority. The Tennessee Valley Authority, which supplies power to almost 9 million Americans, "has not fully implemented appropriate security practices to protect the control systems used to operate its critical infrastructures, leaving them "vulnerahle to disruption," the Govermment Accountability Office found. Simply put, that means skilled hacker could disrupt the system und cause blackout. Rep. James Langevin, Rhode Island Democrat, fears the problem is much larger than just the TVA. "If they not socure, I don't 't have a great deal of confidence that the rest of our critical infrastructure on the electric grid is secure," he said. The TVA operates 52 nuclear, hydroclectric and fossil fuel facilitics in the southeastern United States. Among the government watchdog agency findings: The TVA's firewalls have been bypassed or are inadequately configured Passwords are not effective Servers and work stations lack key patches and effective virus protection Intrusion-dctcction systems are not adequate Some locations lack cnough physical security around control systems. The GAO recommends 73 steps to correct the prohlems in its report to Congress. Seplember, CNN first aired dramatic footage could destroy electrical equipment. The experiment, dubbed " "Aurora," caused generator fall apart and grind to a halt after a computer allack on its control system. The test was conducted by scientists at the Idaho National Laboratory. In October, the North American Electric Reliubility Corp. told Congress that 75 percent of utilities had taken steps to mitigate the Aurora vulnerability. but Langevin said it now appears that Congress was misled. congressional audit of the electric reliability corporation claim cast doubt the assertion that most utilities were tuking steps to fix the problem. "It appeurs that they just made those numbers up. Langevin said. "It is not ucceptable. It is outrageous." He suid the result is there is now no cleur picture of how vulnerable utilities are to eyber attacks. The electric reliability corporation a nongovernmental group that oversees the power system and comprises members CNN that Cooper Industrics the only manufacturer of hardware that can close the Aurora vulnerability. The company estimated it would need to sell about 10,000 devices to fix the problem nationwide. It has sold just over 100, told CNN. Langevin said the foderal government may need new powers to require utilitics to take corrective actions close eyber security gaps, and will press to give those powers the Federal Energy Regulatory Commission. The congressman is chairing Emerging Threats, Cyhersecurity, and Science and Technology subcommitle hearing Wednesday aftemoon. Representatives of the TVA, the GAO, the federal commission and the electric reliability corporation ure to appear before the subcommittec.
report. 2007 President Bush visited the Browns Ferry Nuclear Plant, operated
government experiment demonstrating that a cyber atlack
Article
industry and some consumers told CNN it regrets the confusion. Experts told
(Q: Are eyber attacks real threat? A:]: Yes.
Gold Answer: skilled hacker could disrupt the system and cause a blackout.
IQ: What steps are being taken access vulnerabilities? A:]: The GAO recommends steps. Gold Answer: 73
Q: What percentage of utilities were fixed to combat attacks? A:]: 75 percent.
CTRLsum
Gold Answer: 75 percent
[Q: Who can disrupt the Tennessee Valley Authority? A:]: A skilled hacker.
Gold Answer: skilled hacker
IQ: What was Congress told? A:}: Nothing.
Gold Answer: percent utilities had taken steps to miti the Aurora vudnerability.
34


Reviews:
[{"review_id": "seX3Hn_4roa", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a two-stage summarization system where a document is provided along with (optionally) keywords or a prompt. This supplemental information helps to guide the summarization and possibly make it more user-specific. The keywords and prompt can also be guessed automatically by a BERT-base model, which seems to improve automatic metrics on CNN/daily mail.\n\nStrengths:\n* Moving beyond the conventional 'document/summary' framework of existing summarization approaches is a strength of this paper. This paper studies a few different ways that the summaries can be controlled: through prompts, entities, or one-sentence summaries of summaries (contribution and purpose summarization). These seem novel at least to this reviewer and could be helpful for future work.\n* When using oracle guidance, performance increases on several different datasets for slightly different forms of summarization (CNN/DM, arxiv, bigpatent).\n* The idea of using a two-stage approach (with a BERT-Base extractor to guess keywords to guide the summary) seems novel to this reviewer, and it seems to enable this approach to perform well even in an unconditional setting.\n\nWeaknesses:\n* The main weakness to this reviewer is that the evaluation might not be sufficiently convincing to test the key hypothesis: that these keywords/prompts can enable users to get summaries that are closer to their intent (like Figure 1). To this reviewer, this necessitates a human evaluation. Though testing factual correctness in Table 3 seems like a good start to this reviewer, measuring overall summarization quality (both conditional and unconditional on user intent) through a human evaluation seems necessary.\n* (minor) one possible reason why the two-stage approach might perform better on unconditional summarization is because there are more parameters when ensembling BERT-Base and BART. Possibly doing something multitask within a single BART model might be cleaner and could clearly test whether the gains come from more parameters/computation, or the keyword approach.\n\n\nOverall, to this reviewer, this paper seems like it would be strong if it had human evaluations of summarization quality. I would be willing to raise my score if those were provided.\n\n----\n\nUpdate: thanks for the additional human evaluation results! These help and the results on excluding unimportant entities seem strong to this reviewer. Perhaps it might be more helpful for the annotators themselves to try to interact with the summarizer in some way, but that's a more minor point.\n\nAnyways, I bumped up my score from 5->7.", "extra_scores": {}, "extra_text": {}}, {"review_id": "gg7-miWPeLQ", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Summary:\n\nBuilds/extends on Controllable Abstractive Summarization (Fan et al) using keywords and other prompts. There’s two phases, and both phases are independent:\n\n1. Extract keywords, z,  using a BERT classifier/sequence-tagger trained to predict keywords\n2. Fine-tune BART (Lewis et al) to learn p(summary | document, z).\n\nOne can use automatic keywords using (1) and get uncontrolled generation, for which they present SOTA results on some summarization tasks. Two datasets are collected: (a) intro->contributions from arxiv papers; (b) patent->one-sentence summary, which are used to measure performance of prompts specific to those tasks.\n\n# Pros:\n1. Improves state-of-the-art results on some summarization benchmarks.\n2. Provides BERTScore results in addition to ROUGE.\n3. Results provided across multiple summarization datasets.\n4. Interesting new datasets for measuring document+prompt->summary performance\n5. Interesting zero-shot/transfer results from summarization to Question-answering.\n\n# Cons: \n1. Contribution in methods over Fan et al + BART (Lewis et al) is minimal. Results in Fan et al were weak because the underlying model was much weaker than BART, so it is unclear how much results here improved from simply using BART and adding control tokens from Fan et al, which would be a useful baseline to have that is omitted.\n2. Since the focus of the paper is on controlling generation, more results on this would be informative. It is unclear how well control works when not using oracle words or automatically extracted keywords, i.e. user-controlled. An MTurk experiment evaluating how well control works would be useful in assessing this.\n3. Comparing BART/PEGASUS to BART+BERT-based model is a little unfair since BERT is another large model in the system, i.e. the total amount of compute and number of parameters is much greater. A more fair comparison would be to compare using a smaller BART or PEGASUS model such that the model sizes are comparable. It is unclear whether the proposed system would do better than BART/PEGASUS scaled to the same amount of total parameters.\n4. For the prompt tests, how well does BART/PEGASUS do if the decoder is prompted? This baseline would be useful to have. That is, it’s unclear how the p(y | x, z) improves over using p(y | x) by simply prompting the decoder.\n\n# Clarifications/questions:\n1. For \"CONTRIBUTION AND PURPOSE SUMMARIZATION\", what is the fine-tuning process? Are the models fine-tuned on (paper/patent, keywords)->abstract task before testing on intro->contribution generation? \n2. How are entities randomly selected in the example decodes in the Appendix?\n3. Are any of the prompts used in training or is it zero-shot?\n4. What is zero-shot state-of-the-art on the QA tasks? Please add to Table 5. GPT-3 zero-shot results would also be informative.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Qo366HRkgq_", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose an abstractive document summarization model that can\ngenerate summaries that target a specific set of keywords or prompts. This is\nin contrast to generic summarization models that learn to summarize a document\nbut are difficult to control or direct. The authors propose a straightforward\nway of obtaining keywords from an article similar in spirit to Gerhmann et al.\n2018. Alternatively, \"ground truth\" keywords can be found using a reference\nsummary. In either case, the keywords are prepended to the input document and\na BART model is fine-tuned to generate summaries using both the document and\nkeyword content.\n\nThe authors go on to show how a model trained in such a way, which they refer\nto as CTRLsum, can generate entity-focused summaries, by using an entity name\nas the keyword prefix. Additionally, providing differing numbers of keywords\ncan be used to control the length of the generated summary.  The authors also\nshow that CTRLsum can respond sensibly to prompts, i.e. instead of providing\nkeywords, a question or initial phrase is provided.  Useful summarization\nbehavior can be achieved including zero shot question answering, or\nenumeration of a research paper's contributions or the purpose of an\ninvention.\n\nWhile the paper feels largely like an extension of Keskar et al. 2019, the\nevaluation of the proposed methods is very thorough on a variety of settings\nand domains. I especially enjoyed the break out of entity targeted summaries\nbased on whether the entity occurred in the lead and/or reference summary. The\nuse of prompts to obtain question answering and more focused\ncontributions/purpose summarization was also very interesting.\n\nI would be happy to see this paper accepted to ICLR. This paper offers a\nsimple method of obtaining a variety of focused or targeted summarization\nbehaviors from a BART summarization model. In general, I would like to see\nmore work like this exploring methods of controlling pretrained language\nmodels.  The evaluation of the correctness of the generated utterances\nsuggests that this method provides fairly reliable control.\n\nThere several areas where the paper could improve. The explanation of how\nlength control is achieved was not very clear. It would help to have examples\nlike those shown in the appendix present in the section introducing length\ncontrol. \n\nComparisons are to the standard BART model or to Fan et al. (2018) which\nsimilarly prepend important control information to the input. It would be\ninteresting to see an evaluation that compared CTRLsum to BART with a\nconstrained decoding method, such as dynamic beam allocation [1].\n\nAdditionally, the authors should say more about the differences in entity\ncontrol of their method and Fan et al. 2018, which seem on their face to be\nsimilar.\n\nDid the authors experiment with pairs of entities as entity controls? It would\nbe especially interesting to see whether the model preserves the correct\nrelationship between entities, especially for entities that didn't occur in\nthe same sentences in the original document, e.g. one important entity and one\nunimportant entity.\n\n[1] Matt Post and David Vilar. Fast Lexically Constrained Decoding with Dynamic Beam Allocation for Neural Machine Translation. ACL. 2018.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "5Mc64UkeQFv", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Paper Summary:\n* This paper proposes a framework for controllable summarization, CTRLsum.  It is different from standard summarization models that CTRLsum uses a set of keywords extracted from the source text automatically or descriptive prompts to control the summary.  Experiments with three domains of summarization datasets and five control aspects.\n\nStrengthes:\n* The authors investigated the effectiveness of the proposed model through extensive experiments.\n\nWeaknesses:\n* The proposed method that uses keywords as an additional input text is almost the same as CIT (Saito et al., 2020), and the scores of CTRLsum on the CNNDM dataset reported in Table 7 does not outperformed those of CIT.  Also, it is not novel to use descriptive prompts to control natural language generation.\n   * Saito et al.: Abstractive Summarization with Combination of Pre-trained Sequence-to-Sequence and Saliency Models. CoRR abs/2003.13028 (2020)\n* I think that the author's claim, \"keywords and prompts are complementary\", is not evaluated fully.\n\nQuestions:\n\n* With respect to contribution summarization, did you evaluate CTRLsum(keyword without prompt) and CTRLsum(prompt without keywords)?  The control tokens \"the main contributions of this paper are : ( 1 )\" is far from the keywords used during training, and so I think that the keywords are not effective for contribution summarization.  In fact, BART that uses prompt worked well for contribution summarization.\n\n* Did you evaluate the ablation tests with respect to the special token \"|\" and keyword dropout?\n\n* Can CTRLsum control the generation with multiple aspects (length and entity control, length and QA control, etc.) simultaneously?  The length of  summaries generated by CTRLsum is strongly dependent the number of keywords, and so I think it is difficult to simultaneously control multiple aspects including length control.\n\nUpdate:\nThank you for the answers to my questions and additional experiments. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review:  The paper attempts at controllable summarization in two dimensions: Length, and content. Authors try to achieve this through training data generation approach, where they provide a standard BART model with additional keywords (extracted using a BERT model) in training.

The paper's main motivation on controllable summarization is important and interesting, and despite simplicity, the results are generally positive on multiple datasets.
However, despite positive results, reviewers raised several critical concerns, some of which remained unresolved after reviewer/author discussion period. Examples include concerns regarding lack of methodological novelty over prior work (R1, R2, R4), unfair/incomplete comparisons with prior work (R2, R4, R5), and not evaluating on a real user controlled setting instead of automatic keywords (R1, R4). Although the authors tried addressing human evaluation in their revision, some reviewers remained unconvinced. 
Some quotes from reviewer discussions:

> I'm not convinced the human eval was done properly.

> My concerns are not completely addressed and the score remains unchanged. For human evaluation, I agreed with Reviewer X.

Decision: Reject