SlowGuess/NAACL_2021_OCR huggingface dataset:
How to generate summaries of different styles without requiring corpora in the target styles, or training separate models? We present two novel methods that can be deployed during summary decoding on any pre-trained Transformer-based summarization model. (1) Decoder state adjustment instantly modifies decoder final states with externally trained style scorers, to iteratively refine the output against a target style. (2) Word unit prediction constrains the word usage to impose strong lexical control during generation. In experiments of summarizing with simplicity control, automatic evaluation and human judges both find our models producing outputs in simpler languages while still informative. We also generate news headlines with various ideological leanings, which can be distinguished by humans with a reasonable probability.
Generating summaries with different language styles can benefit readers of varying literacy levels (Chandrasekaran et al., 2020) or interests (Jin et al., 2020). Significant progress has been made in abstractive summarization with large pre-trained Transformers (Dong et al., 2019; Lewis et al., 2020; Zhang et al., 2019; Raffel et al., 2019; Song et al., 2019). However, style-controlled summarization is much less studied (Chandrasekaran et al., 2020), and two key challenges have been identified: (1) lack of parallel data, and (2) expensive (re)training, e.g., separate summarizers must be trained or finetuned for a pre-defined set of styles (Zhang et al., 2018). Both challenges call for inference time methods built upon trained summarization models, to adjust styles flexibly and efficiently.
To address these challenges, we investigate just-in-time style control techniques that can be directly applied to any pre-trained sequence-to-sequence (seq2seq) summarization model. We study two methods that leverage external classifiers to favor
Daily Mail Article: ... [A 16-year-old who was born a girl but identifies as a boy has been granted the opportunity to go through male puberty thanks to hormone treatment.] ... [The transgender boy, who has felt as though he is living in the wrong body since he was a child, has been given permission by a Brisbane-based judge to receive testosterone injections] ...
(a) Decoder State Adjustment: [ Queensland teen has been granted hormone treatment. The 16-year-old was born a girl but identifies as a boy.] ... [A judge has granted the teen permission to receive testosterone injections.] ...
(b) Word Unit Prediction: A 16-year-old who was born a girl has been given the right to go through male puberty. The transgender boy has lived in a female body since he was a ...
Figure 1: Sample summaries generated by our style control methods via (a) adjusting decoder states with a simplicity scorer and (b) predicting simple words to use. Gray texts are produced by BART but removed after decoder state adjustment. Simplified words and their counterparts in the source are highlighted in blue.
the generation of words for a given style. First, decoder state adjustment is proposed to alter the decoder final states with feedback signaled by style scorers, which are trained to capture global property. Second, to offer stronger lexical control, we introduce word unit prediction that directly constrains the output vocabulary. Example system outputs are displayed in Fig. 1. Notably, our techniques are deployed at inference time so that the summary style can be adaptively adjusted during decoding.
We experiment with two tasks: (1) simplicity control for document summarization with CNN/Daily Mail, and (2) headline generation with various ideological stances on news articles from the SemEval task (Kiesel et al., 2019) and a newly curated corpus consisting of multi-perspective stories from AllSides<sup>1</sup>. In this work, the algorithms are experimented with the BART model (Lewis et al., 2020), though they also work with other Transformer models. Both automatic and human
www.allsides.com
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5942-5953 June 6-11, 2021. ©2021 Association for Computational Linguistics
evaluations show that our models produce summaries in simpler languages than competitive baselines, and the informativeness is on par with a vanilla BART. Moreover, headlines generated by our models embody stronger ideological leaning than nontrivial comparisons.[2]
Summarizing documents into different styles are mainly studied on news articles, where one appends style codes as extra embeddings to the encoder (Fan et al., 2018), or connects separate decoders with a shared encoder (Zhang et al., 2018). Similar to our work, Jin et al. (2020) leverage large pre-trained seq2seq models, but they modify model architecture by adding extra style-specific parameters. Nonetheless, existing work requires training new summarizers for different target styles or modifying the model structure. In contrast, our methods only affect decoder states or lexical choices during inference, allowing on-demand style adjustment for summary generation.
Style-controlled text generation has received significant research attentions, especially where parallel data is scant (Lample et al., 2019; Shang et al., 2019; He et al., 2020). Typical solutions involve disentangling style representation from content representation, and are often built upon autoencoders (Hu et al., 2017) with adversarial training objectives (Yang et al., 2018). The target style is then plugged in during generation. Recently, Dathathri et al. (2020) propose plug and play language models (PPLMs) to alter the generation style by modifying all key-value pairs in the Transformer, which requires heavy computation during inference. Krause et al. (2020) then employ a generative discriminator (GeDi) to improve efficiency. Our methods are more efficient since we only modify the decoder final states or curtail the vocabulary.
3 Inference Time Style Control
3.1 Global Characteristic Control via Decoder State Adjustment
Given a style classifier  $q(z|\cdot)$  that measures to which extent does the current generated summary resemble the style  $z$ , we use its estimate to adjust the final decoder layer's state  $\mathbf{o}_t$  at step  $t$  with gradient descent, as illustrated in Fig. 2. The
Figure 2: Just-in-time style control: (1) Decoder state adjustment takes in a style score and iteratively updates  $\mathbf{o}_t$ ; (2) Word unit prediction controls the vocabulary.
output token is produced as  $p(y_{t}|y_{1:t - 1},\mathbf{x}) =$  softmax( $\mathbf{W}_e\mathbf{o}_t$ ),  $\mathbf{W}_e$  is the embedding matrix.
Concretely, to generate the  $t$ -th token, a style score of  $q(z|\mathbf{y}_{1:t + 2})$  is first computed. In addition to what have been generated up to step  $t - 1$ , we also sample  $y_{t}$  and two future tokens for style estimation. The decoder state is updated as follows:
$$
\mathbf {o} _ {t} \leftarrow \mathbf {o} _ {t} - \lambda \nabla_ {\mathbf {o} _ {t}} [ - q (z | \mathbf {y} _ {1: t + 2}) ] \tag {1}
$$
where  $\lambda$  is the step size. Gradient descent is run for 10 iterations for document summarization and 30 iterations for headline generation.
Below, we define one discriminative and one generative style classifier, to illustrate the method.
Discriminative Style Scorer. We feed the tokens into a RoBERTa encoder (Liu et al., 2019) and use the contextualized representation of the BOS token, i.e.,  $\mathbf{h}_0$ , to predict the style score as  $p_{sty}(z|\cdot) = \mathrm{softmax}(\mathbf{W}_s\mathbf{h}_0)$ , where  $\mathbf{W}_{*}$  are learnable parameters in this paper. At step  $t$  of summary decoding, the style score is estimated as:
$$
q (z | \mathbf {y} _ {1: t + 2}) = \log p _ {s t y} (z | \mathbf {y} _ {1: t + 2}) \tag {2}
$$
For the discriminative style scorer, the step size  $\lambda$  is set to 1.0.
Generative Language Model Scorer. We build a class-conditional language model (CC-LM) from texts prepended with special style-indicating tokens. Concretely, the CC-LM yields probabilities  $p_{LM}(y_{t'} | \mathbf{y}_{1:t' - 1}, z)$  ( $p_{LM}(y_{t'}, z)$  for short), conditional on the previously generated tokens  $\mathbf{y}_{1:t' - 1}$  and the style  $z$ . As the summarizer's output probability  $p(y_{t'})$  should be close to the language model's estimate, the style score is defined as:
$$
q (z | \mathbf {y} _ {1: t + 2}) = \frac {1}{t + 2} \sum_ {t ^ {\prime} = 1} ^ {t + 2} p _ {L M} \left(y _ {t ^ {\prime}}, z\right) \log p \left(y _ {t ^ {\prime}}\right) \tag {3}
$$
Here we use a step size  $\lambda$  of 0.1.
2Our code and data are available at: https:// shuyangcao.github.io/projects/inference_ style_control.
3.2 Lexical Control via Word Unit Prediction
Lexical control is another tool for managing summary style, as word choice provides a strong signal of language style. Given an input document, our goal is to predict a set of word units (e.g., the subwords used in BART pre-training) that can be used for summary generation. For instance, if the input contains "affix", we will predict "stick" to be used, while excluding the original word "affix". A similar idea has been used to expedite sequence generation (Hashimoto and Tsuruoka, 2019), though our goal here is to calculate the possibilities of different lexical choices.
Concretely, after encoding the input  $\mathbf{x}$  by RoBERTa, we take the average of all tokens' contextual representations, and pass it through a residual block (He et al., 2016) to get its final representation  $\tilde{\mathbf{R}}$ . We then compute a probability vector for all word units in the vocabulary as  $\mathbf{p}^r = \mathrm{sigmoid}(\mathbf{W}_r\tilde{\mathbf{R}})$ . The top  $v$  word units with the highest probabilities are selected and combined with entity names from the input to form the new vocabulary, from which the summary is generated. We use  $v = 1000$  in all experiments.
Dynamic Prediction. We also experiment with a dynamic version, where the word unit predictor further considers what have been generated up to a given step. In this way, the new vocabulary is updated every  $m$  steps ( $m = 5$  for document summarization, and  $m = 3$  for headline generation).
4 Simplicity-controlled Document Summarization
For experiments, we use BART fine-tuned on the CNN/DailyMail (CNN/DM) (Hermann et al., 2015), by following Lewis et al. (2020) for data preprocessing and splitting. The numbers of data in train, validation and test splits are 287,188, 13,367 and 11,490, respectively.
We use paragraph pairs from normal and simple English Wikipedia articles in Hua and Wang (2019) for simplicity style scorer and class-conditional language model training. We split the pairs into 86,467, 10,778, and 10,788 for training, validation and testing, respectively. On the test set, our simplicity style scorer achieves an F1 score of 89.7 and our class-conditional language model achieves a perplexity of 30.35.
To learn the word unit predictor, for each paragraph pair, the predictor reads in the normal version and is trained to predict the word units used in the
Model	Style	Flu.	Cont.
Simp.↑	%Simp.↑	Rd.↓	PPL↓	BERT↑
BART	56.93	62.70	8.06	34.05	88.62
RERANKING	71.33	62.68	8.04	36.17	88.62
LBLCTRL	56.21	62.71	8.07	28.85	88.57
CTRLGEN	81.56	64.78	7.79	70.36	88.01
TRANS	59.78	63.03	7.99	33.17	88.46
GEDI	71.33	62.57	7.88	33.48	88.79
LIGHTLS	69.02	64.92	7.72	76.37	86.98
Ours w/ Decoder State Adjustment
SIMP. SCORER	86.67	62.94	7.77	34.20	88.71
SIMP. CC-LM	75.04	64.27	7.69	30.49	88.73
Ours w/ Word Unit Prediction
WORDU	95.85	67.23	7.19	27.40	87.76
DYNAMIC WORDU	93.87	67.37	7.23	28.42	87.91
Table 1: Automatic evaluation on summarization with simplicity, with simplicity level by our scorer (Simp., probability multiplied by 100), % of words in the Dale-Chall simple word list (%Simp.), Dale-Chall readability (Rd.), fluency by perplexity (PPL), and content metric by BERTScore (BERT). Our models are significantly better than the comparisons  $(p < 0.005)$  on simplicity and readability, except for CTRLGEN and LIGHTLS.
simple version. For the dynamic version, it predicts which word units are used to generate the rest of the text, after every 5 steps. Recalls for the two predictors on the test set are 81.5 and 80.0.
For comparison, we consider RERANKING beams based on our style score at the last step. We also use a label-controlled (LBLCTRL) baseline as described in Niu and Bansal (2018), where summaries in the training data are labeled as simple or normal by our scorer. We further compare with GEDI and two pipeline models: a style transfer model (Hu et al., 2017) applied on the output of BART (CTRLGEN) and a normal-to-simple translation model fine-tuned from BART (TRANS), both trained on Wikipedia. Finally, we consider LIGHTLS (Glavaš and Štajner, 2015), a rule-based lexical simplification model.
Automatic Evaluation. Table 1 shows that our models' outputs have significantly better simplicity and readability while preserving fluency and a comparable amount of salient content. Key metrics include simplicity level estimated by our scorer and Dale-Chall readability (Chall and Dale, 1995). We use GPT-2 perplexity (Radford et al., 2019) to measure fluency, and BERTScore (Zhang* et al., 2020) for content preservation. Our inference time style control modules can adaptively change the output style, and thus outperform reranking at the end of generation or using pipeline models. More
Model	Inf.↑	Flu.↑	Simp.R.↓	Top 1↑
BART	4.45	4.90	2.19	19.0%
GEDI	4.48	4.83	2.00	23.8%
SIMP. SCORER	4.53	4.83	1.66*	48.4%
DYNAMIC WORDU	4.36	4.84	1.65*	57.9%
Table 2: Human evaluation on informativeness (Inf.), fluency (Flu.), simplicity ranking (Simp.R.), and percentage of summaries ranked as simplest (Top 1). Krippendorff's  $\alpha$ : 0.38, 0.22, and 0.16 (first three metrics).  $*$ : significantly better than comparisons ( $p < 0.005$ ).
over, by iteratively adjusting the decoder states, our methods deliver stronger style control than GEDI, which only adjusts the probability once per step.
When comparing among our models, we find that word unit prediction is more effective at lexical simplification than updating decoder states, as demonstrated by the higher usage of simple words according to the Dale-Chall list. We believe that strong lexical control is achieved by directly pruning output vocabulary, whilst decoder state adjustment is more poised to capture global property, e.g., sentence compression as shown in Fig. 1. Moreover, we compute the edit distance between our style-controlled system outputs and the summaries produced by the fine-tuned BART. We find that adjusting decoder states with style scorer and language model yields an edit distance of 45.7 and 47.4, compared to larger distances of 56.7 and 54.3 given by word unit prediction and with additional dynamic prediction.
Human Evaluation. We recruit three fluent English speakers to evaluate system summaries for informativeness—whether the summary covers important information from the input, and fluency—whether the summary is grammatical, on a scale of 1 (worst) to 5 (best). They then rank the summaries by simplicity level (ties are allowed). 50 samples are randomly selected for evaluation, and system summaries are shuffled. As seen in Table 2, summaries by our models are considered simpler than outputs of BART and GEDI, with better or comparable informativeness.
5 Ideology-controlled Headline Generation
To generate news headlines of various ideological leanings, we use the SemEval Hyperpartisan News Detection dataset (Kiesel et al., 2019), where each article is labeled with a stance: left, leaning left, neutral, leaning right, or right. Here, we combine left and leaning-left articles into one bucket, and
Model	Left	Right
Ideol.	BERT	Ideol.	BERT
BART	18.63	91.03	19.04	91.03
RERANKING	30.80	90.68	30.11	90.66
LBLCTRL	20.59	90.97	20.89	91.02
GEDI	12.64	84.84	3.61	84.84
Ours w/ Decoder State Adjustment
IDEOL. SCORER	31.15	90.08	30.54	90.17
IDEOL. CC-LM	23.74	89.65	20.79	89.65
Ours w/ Word Unit Prediction
WORDU	21.30	89.64	20.42	90.13
DYNAMIC WORDU	21.53	89.49	20.09	90.19
Table 3: Ideological headline generation results. Using ideology scorer to update decoder states yields the highest ideology scores (multiplied by 100).
Figure 3: LIWC word usage changes of "negate" and "affect", compared to neutral headlines. In each subfigure, left and right panels correspond to left and right leaning stances.
similarly for right and leaning-right articles. We use the lead paragraph as the input, and the headline as the target generation. The data is processed following Rush et al. (2015), and split into 346,985 for training, 30,000 each for validation and testing. Details of the ideology distribution for SemEval are in Appendix B.
We fine-tune BART and train ideology classifiers on the SemEval training set. First, two binary style scorers are trained on headlines of left and right stances, with F1 scores of 76.1 and 78.0, respectively. One class-conditional language model is trained on headlines with a stance token (left or right) prepended, achieving a perplexity of 54.7. To learn the word unit predictor for the left (and similarly for the right), we use samples that are labeled as left-leaning, treat the lead paragraph as the input, and then predict the word units used in the headline. Recalls for our predictors range from 77.8 to 83.5.
Automatic Evaluation with SemEval. Table 3 shows that our decoder state adjustment model with the ideology scorer obtains the highest ideology scores, due to its effectiveness at capturing
Model	Rel.	Edit	Hmn	Hmn Acc.
Human	4.01	12.24	60.8%	73.3%
RERANKING	4.71	3.90	24.5%	52.5%
LBLCTRL	4.70	2.30	11.6%	71.4%
IDEOL. SCORER (ours)	4.47	8.86*	42.5%*	53.9%
DYNAMIC WORDU (ours)	4.66	4.20	25.8%	51.6%
the global context—stance is often signaled by the joint selection of entities and sentiments.
One might be interested in which words are favored for ideology-controlled generation. To that end, we analyze the change of word usages with Linguistic Inquiry and Word Count (LIWC) (Pennebaker et al., 2015). In Fig. 3, it can be seen that word unit prediction-based models generate more "negations", consistent with trends observed in human-written headlines. Meanwhile, models with decoder state adjustment and the baselines all use more "affect" words in both stances, indicating that they consider it easier to use explicit sentiments to demonstrate the stances.
Human Evaluation with AllSides. Given the low ideology scores in Table 3, we further study if human can distinguish the stances in human-written and system generated headlines. News clusters from AllSides are used, where each cluster focuses on one story, with multiple paragraph-headline pairs from publishers of left, neutral, and right ideological leanings. We use the lead paragraph as the input, and collect 2,985 clusters with samples written in all three stances. More details of the collection are in Appendix B. We test and report results by using lead paragraphs from neural articles as the input to construct headlines of left and right ideological stances.
We randomly pick 80 samples and include, for each sample, two headlines of different stances generated by each system. Raters first score the relevance of the generated headlines to the neutral paragraph's headline, on a scale of 1 to 5. They then read each pair of headlines to decide whether they are written in different stances, and if so, to label them. Table 4 highlights the intrinsic difficulty of capturing ideological language usage: Even reference headlines are only distinguishable in  $60.8\%$
Paragraph: The Obama administration on Thursday rolled out new efforts aimed at curtailing gun violence ...
Table 4: Human evaluation of ideology-controlled headline generation with relevance (Rel.), edit distance (Edit) between left and right headlines, % of samples perceived as having different stances (Hmn), and (among them) accuracy of identified stances (Hmn Acc.). Krippendorff's  $\alpha$  of relevance: 0.48.  $*$ : significantly better than other models ( $p < 0.005$ ).
REFERENCE
[L]: obama offers new executive actions on gun control
[R]: administration announces new gun control measures, targets military surplus imports
IDEOL.SCORER
[L]: u.s. moves to curb gun violence with new rules
[R]: obama admin to tighten gun control laws
DYNAMIC WORDU
[L]: obama unveils new steps to curb gun violence
[R]: obama administration unveils new gun control measures
Table 5: Sample generated headlines with left (shaded in blue) and right (red) stances. Phrases that are typically used by a stance are in bold.
of the cases, among which the stance identification accuracy is  $73.3\%$ . In comparison,  $42.5\%$  of the output pairs by the decoder state adjustment model can be distinguished, significantly higher than those of the baselines  $(24.5\%$  and  $11.6\%)$ . Sample outputs by our models are shown in Table 5, with more outputs included in Appendix E.
We present two just-in-time style control methods, which can be used in any Transformer-based summarization models. The decoder state adjustment technique modifies decoder final states based on externally trained style scorers. To gain stronger lexical control, word unit prediction directly narrows the vocabulary for generation. Human judges rate our system summaries to be simpler with better readability. We are also able to generate headlines with different ideological leanings.
This research is supported in part by National Science Foundation through Grant IIS-1813341, and by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via contract # FA8650-17-C-9116. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of ODNI, IARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein. We thank all the anonymous reviewers for their constructive suggestions.