SlowGuess/ACL_2020_OCR huggingface dataset:
Di Jin, $^{1}$  Zhijing Jin, $^{2}$  Joey Tianyi Zhou, $^{3*}$  Lisa Orii, $^{4}$  Peter Szolovits $^{1}$
<sup>1</sup>CSAIL, MIT, <sup>2</sup>Amazon Web Services, <sup>3</sup>A*STAR, Singapore, <sup>4</sup>Wellesley College
{jindi15,psz}@mit.edu，zhijing.jin@connect.hku.hk
zhouty@ihpc.a-star.edu.sg, lorii@wellesley.edu
Current summarization systems only produce plain, factual headlines, but do not meet the practical needs of creating memorable titles to increase exposure. We propose a new task, Stylistic Headline Generation (SHG), to enrich the headlines with three style options (humor, romance and clickbait), in order to attract more readers. With no style-specific article-headline pair (only a standard headline summarization dataset and mono-style corpora), our method TitleStylist generates style-specific headlines by combining the summarization and reconstruction tasks into a multitasking framework. We also introduced a novel parameter sharing scheme to further disentangle the style from the text. Through both automatic and human evaluation, we demonstrate that TitleStylist can generate relevant, fluent headlines with three target styles: humor, romance, and clickbait. The attraction score of our model generated headlines surpasses that of the state-of-the-art summarization model by  $9.68\%$ , and even outperforms human-written references. $^{1}$
Every good article needs a good title, which should not only be able to condense the core meaning of the text, but also sound appealing to the readers for more exposure and memorableness. However, currently even the best Headline Generation (HG) system can only fulfill the above requirement yet performs poorly on the latter. For example, in Figure 1, the plain headline by an HG model "Summ: Leopard Frog Found in New York City" is less eye-catching than the style-carrying ones such as "What's That Chuckle You Hear? It May Be the New Frog From NYC."
Figure 1: Given a news article, current HG models can only generate plain, factual headlines, failing to learn from the original human reference. It is also much less attractive than the headlines with humorous, romantic and click-baity styles.
To bridge the gap between the practical needs for attractive headlines and the plain HG by the current summarization systems, we propose a new task of Stylistic Headline Generation (SHG). Given an article, it aims to generate a headline with a target style such as humorous, romantic, and click-baity. It has broad applications in reader-adapted title generation, slogan suggestion, auto-fill for online post headlines, and many others.
SHG is a highly skilled creative process, and usually only possessed by expert writers. One of the most famous headlines in American publications, "Sticks Nix Hick Pix," could be such an example. In contrast, the current best summarization systems are at most comparable to novice writers who provide a plain descriptive representation of the text body as the title (Cao et al., 2018b,a; Lin et al., 2018; Song et al., 2019; Dong et al., 2019). These systems usually use a language generation model that mixes styles with other linguistic patterns and inherently lacks a mechanism to control the style
$^{1}$ Our code is available at https://github.com/
jind11/TitleStylist.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5082-5093
explicitly. More fundamentally, the training data comprise of a mixture of styles (e.g., the Gigaword dataset (Rush et al., 2017)), obstructing the models from learning a distinct style.
In this paper, we propose the new task SHG, to emphasize the explicit control of style in headline generation. We present a novel headline generation model, TitleStylist, to produce enticing titles with target styles including humorous, romantic, and click-baity. Our model leverages a multitasking framework to train both a summarization model on headline-article pairs, and a Denoising Autoencoder (DAE) on a style corpus. In particular, based on the transformer architecture (Vaswani et al., 2017), we use the style-dependent layer normalization and the style-guided encoder-attention to disentangle the language style factors from the text. This design enables us to use the shared content to generate headlines that are more relevant to the articles, as well as to control the style by plugging in a set of style-specific parameters. We validate the model on three tasks: humorous, romantic, and click-baity headline generation. Both automatic and human evaluations show that TitleStylist can generate headlines with the desired styles that appeal more to human readers, as in Figure 1.
The main contributions of our paper are listed below:
- To the best of our knowledge, it is the first research on the generation of attractive news headlines with styles without any supervised style-specific article-headline paired data.
- Through both automatic and human evaluation, we demonstrated that our proposed TitleStylist can generate relevant, fluent headlines with three styles (humor, romance, and clickbait), and they are even more attractive than human-written ones.
- Our model can flexibly incorporate multiple styles, thus efficiently and automatically providing humans with various creative headline options for references and inspiring them to think out of the box.
Our work is related to summarization and text style transfer.
Headline Generation as Summarization
Headline generation is a very popular area of research. Traditional headline generation methods mostly focus on the extractive strategies using linguistic features and handcrafted rules (Luhn, 1958; Edmundson, 1964; Mathis et al., 1973; Salton et al., 1997; Jing and McKeown, 1999; Radev and McKeown, 1998; Dorr et al., 2003). To enrich the diversity of the extractive summarization, abstractive models were then proposed. With the help of neural networks, Rush et al. (2015) proposed attention-based summarization (ABS) to make Banko et al. (2000)'s framework of summarization more powerful. Many recent works extended ABS by utilizing additional features (Chopra et al., 2016; Takase et al., 2016; Nallapati et al., 2016; Shen et al., 2016, 2017a; Tan et al., 2017; Guo et al., 2017). Other variants of the standard headline generation setting include headlines for community question answering (Higurashi et al., 2018), multiple headline generation (Iwama and Kano, 2019), user-specific generation using user embeddings in recommendation systems (Liu et al., 2018), bilingual headline generation (Shen et al., 2018) and question-style headline generation (Zhang et al., 2018a).
Only a few works have recently started to focus on increasing the attractiveness of generated headlines (Fan et al., 2018; Xu et al., 2019). Fan et al. (2018) focuses on controlling several features of the summary text such as text length, and the style of two different news outlets, CNN and DailyMail. These controls serve as a way to boost the model performance, and the CNN- and DailyMail-style control shows a negligible improvement. Xu et al. (2019) utilized reinforcement learning to encourage the headline generation system to generate more sensational headlines via using the readers' comment rate as the reward, which however cannot explicitly control or manipulate the styles of headlines. Shu et al. (2018) proposed a style transfer approach to transfer a non-clickbait headline into a clickbait one. This method requires paired news articles-headlines data for the target style; however, for many styles such as humor and romance, there are no available headlines. Our model does not have this limitation, thus enabling transferring to many more styles.
Text Style Transfer
Our work is also related to text style transfer, which aims to change the style attribute of the text while
preserving its content. First proposed by Shen et al. (2017b), it has achieved great progress in recent years (Xu et al., 2018; Lample et al., 2019; Zhang et al., 2018b; Fu et al., 2018; Jin et al., 2019; Yang et al., 2018; Jin et al., 2020). However, all these methods demand a text corpus for the target style; however, in our case, it is expensive and technically challenging to collect news headlines with humor and romance styles, which makes this category of methods not applicable to our problem.
The model is trained on a source dataset  $S$  and target dataset  $T$ . The source dataset  $S = \{(\pmb{a}^{(i)},\pmb{h}^{(i)})\}_{i=1}^{N}$  consists of pairs of a news article  $\pmb{a}$  and its plain headline  $\pmb{h}$ . We assume that the source corpus has a distribution  $P(A,H)$ , where  $A = \{\pmb{a}^{(i)}\}_{i=1}^{N}$ , and  $H = \{\pmb{h}^{(i)}\}_{i=1}^{N}$ . The target corpus  $T = \{\pmb{t}^{(i)}\}_{i=1}^{M}$  comprises of sentences  $\pmb{t}$  written in a specific style (e.g., humor). We assume that it conforms to the distribution  $P(T)$ .
Note that the target corpus  $T$  only contains style-carrying sentences, not necessarily headlines — it can be just book text. Also no sentence  $\pmb{t}$  is paired with a news article. Overall, our task is to learn the conditional distribution  $P(T|A)$  using only  $S$  and  $T$ . This task is fully unsupervised because there is no sample from the joint distribution  $P(A,T)$ .
3.2 Seq2Seq Model Architecture
For summarization, we adopt a sequence-to-sequence (Seq2Seq) model based on the Transformer architecture (Vaswani et al., 2017). As in Figure 2, it consists of a 6-layer encoder  $E(\cdot; \theta_E)$  and a 6-layer decoder  $G(\cdot; \theta_G)$  with a hidden size of 1024 and a feed-forward filter size of 4096. For better generation quality, we initialize with the MASS model (Song et al., 2019). MASS is pretrained by masking a sentence fragment in the encoder, and then predicting it in the decoder on large-scale English monolingual data. This pretraining is adopted in the current state-of-the-art systems across various summarization benchmark tasks including HG.
3.3 Multitask Training Scheme
To disentangle the latent style from the text, we adopt a multitask learning framework (Luong et al., 2015), training on summarization and DAE simultaneously (as shown in Figure 3).
Figure 2: The Transformer-based architecture of our model.
Figure 3: Training scheme. Multitask training is adopted to combine the summarization and DAE tasks.
Supervised Seq2Seq Training for  $E_S$  and  $G_S$  With the source domain dataset  $S$ , based on the encoder-decoder architecture, we can learn the conditional distribution  $P(H|A)$  by training  $z_S = E_S(A)$  and  $H_S = G_S(z_S)$  to solve the supervised Seq2Seq learning task, where  $z_S$  is the learned latent representation in the source domain. The loss function of this task is
$$
\mathcal {L} _ {S} \left(\boldsymbol {\theta} _ {\boldsymbol {E} _ {S}}, \boldsymbol {\theta} _ {\boldsymbol {G} _ {S}}\right) = \mathbb {E} _ {(\boldsymbol {a}, \boldsymbol {h}) \sim S} \left[ - \log p (\boldsymbol {h} | \boldsymbol {a}; \boldsymbol {\theta} _ {\boldsymbol {E} _ {S}}, \boldsymbol {\theta} _ {\boldsymbol {G} _ {S}}) \right], \tag {1}
$$
where  $\theta_{E_S}$  and  $\theta_{G_S}$  are the set of model parameters of the encoder and decoder in the source domain and  $p(h|a)$  denotes the overall probability of generating an output sequence  $h$  given the input article  $a$ , which can be further expanded as follows:
$$
p \left(\boldsymbol {h} \mid \boldsymbol {a}; \boldsymbol {\theta} _ {\boldsymbol {E} _ {\boldsymbol {S}}}, \boldsymbol {\theta} _ {\boldsymbol {G} _ {\boldsymbol {S}}}\right) = \prod_ {t = 1} ^ {L} p \left(h _ {t} \mid \left\{h _ {1}, \dots , h _ {t - 1} \right\}, \boldsymbol {z} _ {\boldsymbol {S}}; \boldsymbol {\theta} _ {\boldsymbol {G} _ {\boldsymbol {S}}}\right), \tag {2}
$$
where  $L$  is the sequence length.
DAE Training for  $\theta_{E_T}$  and  $\theta_{G_T}$  For the target style corpus  $T$ , since we only have the sentence  $\pmb{t}$  without paired news articles, we train  $\pmb{z}_{\pmb{T}} = E_{T}(\tilde{\pmb{t}})$  and  $\pmb{t} = G_{T}(\pmb{z}_{\pmb{T}})$  by solving an unsupervised re
construction learning task, where  $z_{T}$  is the learned latent representation in the target domain, and  $\tilde{t}$  is the corrupted version of  $t$  by randomly deleting or blanking some words and shuffling the word orders. To train the model, we minimize the reconstruction error  $\mathcal{L}_T$ :
$$
\mathcal {L} _ {T} (\pmb {\theta} _ {\pmb {E} _ {T}}, \pmb {\theta} _ {\pmb {G} _ {T}}) = \mathbb {E} _ {\pmb {t} \sim \pmb {T}} [ - \log p (\pmb {t} | \tilde {\pmb {t}}) ], \qquad (3)
$$
where  $\theta_{E_T}$  and  $\theta_{G_T}$  are the set of model parameters for the encoder and generator in the target domain. We train the whole model by jointly minimizing the supervised Seq2Seq training loss  $\mathcal{L}_S$  and the unsupervised denoised auto-encoding loss  $\mathcal{L}_T$  via multitask learning, so the total loss becomes
$$
\begin{array}{l} \mathcal {L} \left(\boldsymbol {\theta} _ {\boldsymbol {E} _ {S}}, \boldsymbol {\theta} _ {\boldsymbol {G} _ {S}}, \boldsymbol {\theta} _ {\boldsymbol {E} _ {T}}, \boldsymbol {\theta} _ {\boldsymbol {G} _ {T}}\right) = \lambda \mathcal {L} _ {S} \left(\boldsymbol {\theta} _ {\boldsymbol {E} _ {S}}, \boldsymbol {\theta} _ {\boldsymbol {G} _ {S}}\right) \\ + (1 - \lambda) \mathcal {L} _ {T} \left(\boldsymbol {\theta} _ {\boldsymbol {E} _ {T}}, \boldsymbol {\theta} _ {\boldsymbol {G} _ {T}}\right), \\ \end{array}
$$
where  $\lambda$  is a hyper-parameter.
3.4 Parameter-Sharing Scheme
More constraints are necessary in the multitask training process. We aim to infer the conditional distribution as  $P(T|A) = G_T(E_S(A))$ . However, without samples from  $P(A,T)$ , this is a challenging or even impossible task if  $E_S$  and  $E_T$ , or  $G_S$  and  $G_T$  are completely independent of each other. Hence, we need to add some constraints to the network by relating  $E_S$  and  $E_T$ , and  $G_S$  and  $G_T$ . The simplest design is to share all parameters between  $E_S$  and  $E_T$ , and apply the same strategy to  $G_S$  and  $G_T$ . The intuition behind this design is that by exposing the model to both summarization task and style-carrying text reconstruction task, the model would acquire some sense of the target style while summarizing the article. However, to encourage the model to better disentangle the content and style of text and more explicitly learn the style contained in the target corpus  $T$ , we share all parameters of the encoder between two domains, i.e., between  $E_S$  and  $E_T$ , whereas we divide the parameters of the decoder into two types: style-independent parameters  $\theta_{\mathrm{ind}}$  and style-dependent parameters  $\theta_{\mathrm{dep}}$ . This means that only the style-independent parameters are shared between  $G_S$  and  $G_T$  while the style-dependent parameters are not. More specifically, the parameters of the layer normalization and encoder attention modules are made style-dependent as detailed below.
Type 1. Style Layer Normalization Inspired by previous work on image style transfer (Dumoulin
et al., 2016), we make the scaling and shifting parameters for layer normalization in the transformer architecture un-shared for each style. This style layer normalization approach aims to transform a layer's activation  $x$  into a normalized activation  $z$  specific to the style  $s$ :
$$
z = \gamma_ {s} \left(\frac {\boldsymbol {x} - \mu}{\sigma}\right) - \beta_ {s}, \tag {5}
$$
where  $\mu$  and  $\sigma$  are the mean and standard deviation of the batch of  $\pmb{x}$ , and  $\gamma_{s}$  and  $\beta_{s}$  are style-specific parameters learned from data.
Specifically, for the transformer decoder architecture, we use a style-specific self-attention layer normalization and final layer normalization for the source and target domains on all six decoder layers.
Type 2. Style-Guided Encoder Attention Our model architecture contains the attention mechanism, where the decoder infers the probability of the next word not only conditioned on the previous words but also on the encoded input hidden states. The attention patterns should be different for the summarization and the reconstruction tasks due to their different inherent nature. We insert this thinking into the model by introducing the style-guided encoder attention into the multi-head attention module, which is defined as follows:
$$
Q = \operatorname {q u e r y} \cdot W _ {q} ^ {s} \tag {6}
$$
$$
\boldsymbol {K} = \mathbf {k e y} \cdot \boldsymbol {W} _ {k} \tag {7}
$$
$$
\boldsymbol {V} = \text {v a l u e} \cdot \boldsymbol {W} _ {\boldsymbol {v}} \tag {8}
$$
$$
\operatorname {A t t} (\boldsymbol {Q}, \boldsymbol {K}, \boldsymbol {V}) = \operatorname {S o f t m a x} \left(\frac {\boldsymbol {Q} \boldsymbol {K} ^ {\mathrm {t r}}}{\sqrt {d _ {\mathrm {m o d e l}}}}\right) \boldsymbol {V}, \tag {9}
$$
where query, key, and value denote the triple of inputs into the multi-head attention module;  $W_{q}^{s}$ ,  $W_{k}$ , and  $W_{v}$  denote the scaled dot-product matrix for affine transformation;  $d_{\mathrm{model}}$  is the dimension of the hidden states. We specialize the dot-product matrix  $W_{q}^{s}$  of the query for different styles, so that  $Q$  can be different to induce diverse attention patterns.
We compile a rich source dataset by combining the New York Times (NYT) and CNN, as well as three target style corpora on humorous, romantic, and click-baity text. The average sentence length in
the NYT, CNN, Humor, Romance, and Clickbait datasets are 8.8, 9.2, 12.6, 11.6 and 8.7 words, respectively.
4.1.1 Source Dataset
The source dataset contains news articles paired with corresponding headlines. To enrich the training corpus, we combine two datasets: the New York Times (56K) and CNN (90K). After combining these two datasets, we randomly selected 3,000 pairs as the validation set and another 3,000 pairs as the test set.
We first extracted the archival abstracts and headlines from the New York Times (NYT) corpus (Sandhaus, 2008) and treat the abstracts as the news articles. Following the standard preprocessing procedures (Kedzie et al., 2018), we filtered out advertisement-related articles (as they are very different from news reports), resulting in 56,899 news abstracts-headlines pairs.
We then add into our source set the CNN summarization dataset, which is widely used for training abstractive summarization models (Hermann et al., 2015).<sup>3</sup> We use the short summaries in the original dataset as the news abstracts and automatically parsed the headlines for each news from the dumped news web pages,<sup>4</sup> and in total collected 90,236 news abstract-headline pairs.
4.1.2 Three Target Style Corpora
Humor and Romance For the target style datasets, we follow (Chen et al., 2019) to use humor and romance novel collections in BookCorpus (Zhu et al., 2015) as the Humor and Romance datasets. We split the documents into sentences, tokenized the text, and collected 500K sentences as our datasets.
Clickbait We also tried to learn the writing style from the click-baity headlines since they have shown superior attraction to readers. Thus we used The Examiner - SpamClickBait News dataset, denoted as the Clickbait dataset. We collected 500K headlines for our use.
Some examples from each style corpus are listed in Table 1.
$^{2}$ https://github.com/kedz/  
summmarization-datasets  
 $^{3}$ We use CNN instead of the DailyMail dataset since DailyMail headlines are very long and more like short summaries.  
 $^{4}$ https://cs.nyu.edu/~kcho/DMQA/  
 $^{5}$ https://www.smashwords.com/  
 $^{6}$ https://www.kaggle.com/therohk/  
examine-the-examiner
Style	Examples
Humor	- The crowded beach like houses in the burbs and the line ups at Walmart.
- Berthold stormed out of the brewing argument with his violin and bow and went for a walk with it to practice for the much more receptive polluted air.
Romance	- “I can face it joyously and with all my heart, and soul!” she said.
- With bright blue and green buttercream scales, sparkling eyes, and purple candy melt wings, it sat majestically on a rocky ledge made from chocolate.
Clickbait	- 11-Year-Old Girl and 15-Year-Old Boy Accused of Attempting to Kill Mother: Who Is the Adult?
- Chilly, Dry Weather Welcomes 2010 to South Florida
- End Segregation in Alabama-Bryce Hospital Sale Offers a Golden Opportunity
Table 1: Examples of three target style corpora: humor, romance, and clickbait.
We compared the proposed TitleStylist against the following five strong baseline approaches.
Neural Headline Generation (NHG) We train the state-of-the-art summarization model, MASS (Song et al., 2019), on our collected news abstracts-headlines paired data.
Gigaword-MASS We test an off-the-shelf headline generation model, MASS from (Song et al., 2019), which is already trained on Gigaword, a large-scale headline generation dataset with around 4 million articles.7
Neural Story Teller (NST) It breaks down the task into two steps, which first generates headlines from the aforementioned NHG model, then applies style shift techniques to generate style-specific headlines (Kiros et al., 2015). In brief, this method uses the Skip-Thought model to encode a sentence into a representation vector and then manipulates its style by a linear transformation. Afterward, this transformed representation vector is used to initialize a language model pretrained on a style-specific corpus so that a stylistic headline can be generated. More details of this method can refer to the official website.[8]
$^{7}$ https://github.com/harvardnlp/ sent-summary  $^{8}$ https://github.com/ryankiros/neural-storyteller
Fine-Tuned We first train the NHG model as mentioned above, then further fine-tuned it on the target style corpus via DAE training.
Multitask We share all parameters between  $E_{S}$  and  $E_{T}$ , and between  $G_{S}$  and  $G_{T}$ , and trained the model on both the summarization and DAE tasks. The model architecture is the same as NHG.
To evaluate the performance of the proposed TitleStylist in generating attractive headlines with styles, we propose a comprehensive twofold strategy of both automatic evaluation and human evaluation.
4.3.1 Setup of Human Evaluation
We randomly sampled 50 news abstracts from the test set and asked three native-speaker annotators for evaluation to score the generated headlines. Specifically, we conduct two tasks to evaluate on four criteria: (1) relevance, (2) attractiveness, (3) language fluency, and (4) style strength. For the first task, the human raters are asked to evaluate these outputs on the first three aspects, relevance, attractiveness, and language fluency on a Likert scale from 1 to 10 (integer values). For relevance, human annotators are asked to evaluate how semantically relevant the headline is to the news body. For attractiveness, annotators are asked how attractive the headlines are. For fluency, we ask the annotators to evaluate how fluent and readable the text is. After the collection of human evaluation results, we averaged the scores as the final score. In addition, we have another independent human evaluation task about the style strength – we present the generated headlines from TitleStylist and baselines to the human judges and let them choose the one that most conforms to the target style such as humor. Then we define the style strength score as the proportion of choices.
4.3.2 Setup of Automatic Evaluation
Apart from the comprehensive human evaluation, we use automatic evaluation to measure the generation quality through two conventional aspects: summarization quality and language fluency. Note that the purpose of this two-way automatic evaluation is to confirm that the performance of our model is in an acceptable range. Good automatic evaluation performances are necessary proofs to compliment human evaluations on the model effectiveness.
Summarization Quality We use the standard automatic evaluation metrics for summarization with the original headlines as the reference: BLEU (Papineni et al., 2002), METEOR (Denkowski and Lavie, 2014), ROUGE (Lin, 2004) and CIDEr (Vedantam et al., 2015). For ROUGE, we used the Files2ROUGE<sup>9</sup> toolkit, and for other metrics, we used the pycocoeval toolkit.<sup>10</sup>