Venue: NAACL 2021

# Ad Headline Generation using Self-Critical Masked Language Model

 Yashal Shakti Kanungo

yashalk@amazon.com &Sumit Negi

suminegi@amazon.com &Aruna Rajan

rajarna@amazon.com

###### Abstract

For any E-commerce website it is a nontrivial problem to build enduring advertisements that attract shoppers. It is hard to pass the creative quality bar of the website, especially at a large scale. We thus propose a programmatic solution to generate product advertising headlines using retail content. We propose a state of the art application of Reinforcement Learning (RL) Policy gradient methods on Transformer Vaswani et al. (2017) based Masked Language Models Devlin et al. (2019). Our method creates the advertising headline by jointly conditioning on multiple products that a seller wishes to advertise. We demonstrate that our method outperforms existing Transformer and LSTM + RL methods in overlap metrics and quality audits. We also show that our model-generated headlines outperform human submitted headlines in terms of both grammar and creative quality as determined by audits.

## 1 Introduction

There are a various types of ads. A set of example ads that showcase products selected by sellers along with headlines that advertise them are shown in Figure 1. Sellers create multiple ad campaigns for multiple products, bid in an auction to advertise and pay for clicks on the ad.

An E-Commerce product catalog may have millions of products which can be advertised. To ease the ad headline writing process, humans resort to programmatically padding keywords, or repasting the retail catalog content in the advertisement.

Templated creatives such as "Save Now on ..." or "Buy more (product) of (brand)" save the creative effort but fail to create any excitement or brand identity in the minds of shoppers. High quality headlines are more attractive to shoppers and offer better value proposition. In this paper, we describe how we built a Natural Language Generation (NLG) system to generate instantaneous, attractive and brand identity building headlines for advertisements that intend to promote a wide range of products offered by a brand.

The content associated with a retail product has challenging characteristics. Some product titles have poor structure, grammatical issues, or partial phrases. The product titles also include varying number of product features such as _"Hyper Tough 18V Cordless Drill, 3/8 inch Chuck, Variable Speed, with 1.2Ah Nickel Cadmium Battery, Charger, Bit Holder LED Light"_ along with titles such as _"ZIPIT Grillz Backpack, Cano Grey"_.

The generated headlines need to capture the information present in the retail attributes and at the same time be different and uniquely attractive. Advertisers select multiple related products that are advertised as part of a single ad campaign. The ad campaign headline is then shared across all of these related products. Thus, the headline also needs to generalize the shared characteristics of the products and cannot be specific to a single product within the campaign.

The key contributions of our work are:

* We use Masked Language Model (MLM) for the generation of advertisement headlines using multiple products at the same time. Extensive test-set metrics, quality and grammar audits show that the proposed model outperforms all the baselines and the human-submitted headlines in terms of quality and grammar.
* The novel usage of RL for the training of MLM allows us to directly optimize the MLM for improved headline quality metrics without changing inference setup or latency. Our method can also be applied to any other NLG task such as summarization, translation etc.
* Our model reduces the extensive effort and time that is required to manually create headlines and has low latency.





## 2 Related Work

Natural Language Understanding (NLU) using Language Models (LM) has observed great leaps in recent years. LMs have evolved from using word level models Joulin et al. (2016) to to a variety of extensions to the Transformer Vaswani et al. (2017). The BERT Devlin et al. (2019) employs Transformer in a pre-training setting and introduced the MLM training objective.

Ramachandran et al. Ramachandran et al. (2016) first demonstrated textual generation by using auto-regressive prediction in a seq2seq architecture. Transformer based auto-regressive methods such as GPT2 Radford et al. (2019) and BART Lewis et al. (2019) which predict one word at a time have also shown good results. Zhu et al. Zhu et al. (2020) concatenated BERT representations with the Encoder and Decoder layers of another LM to incorporate pre-trained LM. Another model Dong et al. (2019) combines BERT-based Transformer Encoder with attention masking from the Transformer decoder. Rothe et al. Rothe et al. (2019) combined pre-trained BERT Encoder with GPT decoder for NLG.

Ranzato et al. Ranzato et al. (2016) framed NLG as an RL problem and the generation quality as a reward. The Self-Critical Sequence Training (SCST) approach Rennie et al. (2017) replaces the learned baseline from other approaches Bahdanau et al. (2017) with the model's own inference time algorithm to normalize the rewards.

For advertising, recent works Xu et al. (2019); Hughes et al. (2019) have combined LSTM based pointer network See et al. (2017) with RL methods to generate advertisement headlines. While these methods improve the results, they fail to utilize extensive pre-training of Transformer based models and their various well-demonstrated advantages.

Our method extends BERT based generation Dong et al. (2019) by using Self-Critical policy gradient method Rennie et al. (2017) and jointly conditioning the generated sentence on multiple products at the same time. This allows us to use pre-trained BERT based LMs that can be trained to optimize various inference time metrics that are typically non-differentiable such as BLEU, Rouge, Readability etc.

## 3 Self-Critical Masked Language Model

### Masked Language Model

The BERT model takes an unlabeled input sequence \(x=(x_{1},x_{2},...,x_{|x|})\) and randomly masks some positions \(M_{x}\) by replacing them with a special mask token [MASK], to produce a sequence like \((x_{1},\text{[MASK]},...,x_{|x|})\). All the tokens are embedded and added to special positional embeddings. It then uses \(N\) identical Transformer layers to generate contextualized representation, with each layer employing self-attention by taking in the output of the previous layer. To compute self-attention, the output of the previous layer is projected into triplets of vectors named Query, Key and Value (\(Q,K,V\)) of dimensions \(d\). The attention \(A\) is then given as:

\[A=\text{softmax}(\frac{QK^{T}}{\sqrt{d}})V \tag{1}\]

After the final Transformer layer the model uses a feed forward layer followed by a softmax over the vocabulary to predict the masked tokens. The MLM loss for the sequence \(x\) is then calculated as:

\[\mathcal{L}_{MLM}=-\log\prod_{m\in M_{x}}p(x_{m}|(x_{m^{\prime}}\in x\setminus M _{x})) \tag{2}\]

where \((x_{m^{\prime}}\in x\setminus M_{x})\) represents all the tokens in \(x\) that are not masked and \(m\in M_{x}\) are all the masked positions.

Figure 1: Examples of different product ads from multiple websites across the internet. A variety of ad headlines accompany the products in these ads.





### Encoding multiple products and common headline for Proposed MLM

During training, for a given advertising campaign, our model takes as input it's headline \(x^{h}=(x_{1}^{h},...,x_{|x^{k}|}^{h})\) and a set \(P\) of one or more products. Each product \(p\) is represented by its title \(x^{p}=(x_{1}^{p},...,x_{|xp|}^{p})\). The titles and the headline are tokenized to sub-word tokens.

To encode using the model that only accepts a single product, we simply append '[EOS]' \(\in\mathbb{V}\) to both the title and the headline and concatenate their tokens. The entire concatenated sequence is prepended with '[SOS]' \(\in\mathbb{V}\).

We encode multiple products by concatenating the tokens from different products using a special token '[P_SEP]' \(\in\mathbb{V}\). We replace a token '[UNUSED_0]' \(\in\mathbb{V}\) that remains unused during pre-training, with this special token during multi-product fine-tuning. This makes a distinction between different titles as well as the source and target sub-sequences. It also yields individual embeddings for each product for other tasks.

Only the tokens from the headline \(x^{h}\) are randomly masked with token '[MASK]' \(\in\mathbb{V}\). We discuss results for the model that additionally also masks the source tokens in section 5.1.

The complete process for an example such that all products in the ad have two tokens and the headline has 4 tokens is illustrated in Figure 2.

We also experimented with adding of category based embeddings. The category labels for each product such as "Cell Phones and Accessories" are tokenized to subword units, encoded using the same embedding matrix as that of the title tokens, averaged and added to the title token embeddings.

### Generation using Self-Critical Masked Language Model

The BERT MLM framework with multi-directional attention discussed in Section 3.1 cannot be used for auto-regressive generation directly. This is because, during training, the masked headline words may condition on the future words which are not available during auto-regressive inference. For MLM auto-regressive generation, we employ masked attention Dong et al. (2019) that modifies the attention from equation 1 as below:

\[A_{masked}=\text{softmax}(\frac{QK^{T}}{\sqrt{d}}+\Phi_{ij})V \tag{3}\]

where \(\Phi_{ij}\) represents the attention mask between the positions \(i\) and \(j\). The elements are set to 0 if attention is allowed and \(-\infty\) if it is not allowed. Figure 3 illustrates the attention mask for headline generation using multiple input products.

The BERT MLM uses log-likelihood (Equation 2) of masked words during training to optimize the model parameters. The likelihood is predicted using other ground-truth words during training and other predicted words during inference. This causes exposure bias Ranzato et al. (2016); Rennie et al. (2017) and accumulates error during inference. Moreover, the training is optimized for log-likelihood, while we actually care about other more evolved measures of headline quality such as overlap metrics BLEU Papineni et al. (2002) and ROUGE Lin (2004).

To overcome these issues and improve the quality of the generated headlines, we frame the MLM

Figure 2: The sub-tokens from the product titles and headline are embedded and added with other embeddings that encode the positional and segment information. We also optionally add an embedding that represents the category of the product. During training, the masked tokens are predicted using Transformer layers and the cross-entropy (Eq. 2) loss and Self-Critical (Eq. 9) gradient is used to optimize the model. During inference, we predict one word at a time (left-to-right) in an auto-regressive manner using Beam Search.



as an RL problem. The model is an 'agent' that takes the 'action' of predicting masked words and updates the 'state' such as the self-attention weights. The MLM follows a policy \(\pi_{\theta}\) defined by the parameters \(\theta\) of the model. It receives a reward that is proportional to the quality of the generated headline. This quality may either be the overlap with ground truth headlines that have been approved by internal subject-matter-experts or be predicted by another model. Our goal is to maximize the reward corresponding to a generated headline \(\hat{x}^{h}\) during training, with the tokens at some masked positions \(M_{x^{h}}\) sampled from the model.

We thus minimize the negative expected reward defined by any reward function \(r(\cdot)\) for headline quality \(r(\hat{x}^{h})\) as:

\[\mathcal{L}_{RL}=-\mathbb{E}_{\hat{x}^{h}\sim\pi_{\theta}}[r(\hat{x}^{h})] \tag{4}\]

We can compute the gradient \(\nabla_{\theta}\mathcal{L}_{RL}\) using the REINFORCE algorithm Williams (1992). It is defined as:

\[\nabla_{\theta}\mathcal{L}_{RL}=-\mathbb{E}_{\hat{x}^{h}\sim\pi_{\theta}}[r( \hat{x}^{h})\nabla_{\theta}P] \tag{5}\]

where,

\[P=\sum_{m\in M_{\hat{x}^{h}}}\log p_{\theta}(\hat{x}^{h}_{m}|(\hat{x}^{h}_{ m^{\prime}}\in\hat{x}^{h}\setminus M_{\hat{x}^{h}}) \tag{6}\]

such that \(M_{\hat{x}^{h}}\) are the masked positions and \(\hat{x}^{h}\setminus M_{\hat{x}^{h}}\) are all the unmasked tokens.

To reduce the variance without changing the expected gradient, the algorithm proposes to use a baseline \(b\) that does not depend on the generated headline \(\hat{x}^{h}\). \(b\) is used to normalize the reward along with \(P\) from equation 6 as:

\[\nabla_{\theta}\mathcal{L}_{RL}=-\mathbb{E}_{r(\hat{x}^{h})\sim\pi_{\theta}}[ (r(\hat{x}^{h})-b)\nabla_{\theta}P] \tag{7}\]

A single Monte-Carlo sample for each set of products and headline can be used to approximate the gradient. Using the definition of \(P\) from equation 6, we have the approximate gradient:

\[\nabla_{\theta}\mathcal{L}_{RL}\approx-(r(\hat{x}^{h})-b)\nabla_{\theta}P \tag{8}\]

Instead of using other models to estimate the expected baseline reward Ranzato et al. (2016); Bahdanau et al. (2017), we employ Self-Critical training Rennie et al. (2017) that involves generating two headlines using the same underlying MLM. The first headline \(\hat{x}^{h}\) is generated by sampling from the vocabulary distributions generated by the model for the masked tokens. The second headline \(\hat{z}^{h}\) is generated using the inference time strategy, which uses the token with the maximum probability at each step rather than sampling. The difference in the reward achieved by these two headlines is used to compute the gradient:

\[\nabla_{\theta}\mathcal{L}_{SC\_MLM}\approx-(r(\hat{x}^{h})-r(\hat{z}^{h})) \nabla_{\theta}P \tag{9}\]

where \(P\) is defined by equation 6.

Thus, this method maximizes both the reward of the headlines generated by MLM and the likelihood of correct words by incorporating both the likelihood and the reward in the loss function.

### Inference

During inference, we generate the headline auto-regressively using beam search until we reach the predetermined max length or each beam generates the end token. We have employed a modified version of Length Normalization Wu et al. (2016) to better adapt to our headline lengths and training setup. This is necessary as the default beam search setup uses the log probability of each word to select the best headline. However, this biases the results as longer headlines would have lower probability of

Figure 3: Masked attention partially restricts attention for some token pairs. It prevents attention to headline tokens that would not be accessible during each step of generation during inference.

 

generation. We thus use the following normalized scores for each word to select the best headline:

\[\text{score}(\hat{x}_{i}^{h})=\text{log-likelihood}(\hat{x}_{i}^{h})* \frac{(2+1)^{\alpha}}{(2+i)^{\alpha}} \tag{10}\]

where \(\alpha\) is the length normalization coefficient and \(\hat{x}_{i}^{h}\) is the \(i^{th}\) word of the generated headline in each beam. We also include additional Regular Expression based post-processing to remove extra spaces around various symbols such as '-,+()' etc.

## 4 Experiments

### Training and Inference

We used over 500,000 ad campaigns that were created on Amazon by sellers who have signed-up for advertising. Each campaign contains a set of related products along with an ad headline. We only selected the campaigns that contained English headlines and products with English titles. They were also de-duplicated to only have unique products-headline pairs. The mean product title length is 19.6 words and the mean headline length is 6.16 words. The entire dataset was divided into train (85%), validation (5%) and test (10%) sets. For training, we only selected the campaigns that comply with ad policies as verified by internal experts.

We use the HuggingFace Wolf et al. (2020) implementation of the Transformer BERT 'Large' models as the base for our experiments. The models are pre-trained on WikiPedia and BookCorpus Devlin et al. (2019); Dong et al. (2019). We first fine-tune the pre-trained model for up-to 15 epochs with early stopping using \(\mathcal{L}_{MLM}\) and Adam Kingma and Ba (2014). We then further fine-tune the model for another 15 epochs with early stopping using Adam with \(\nabla\mathcal{L}_{SC\_MLM}\)Equation9. We use the Rouge L F1 Lin (2004) overlap with the approved headlines as the headline quality reward. For a fair comparison, the MLM-only model is fine-tuned for upto 30 epochs.

The model training is very time expensive with a single fine-tuning sub-experiment of 30 epochs taking over 20 days on an Nvidia v100. We thus only performed the essential experiments that help to determine the contribution of different sub-experiments and proposals. We estimated post-experiment that a single fine-tuning sub-experiment of 30 epochs would consume approximately 150 kWh of energy based on the GPU's power draw.

### Baseline

We used a Pointer Network See et al. (2017) based bi-LSTM with intra-decoder and temporal attention. We also used Self-Critical training with the bi-LSTM, similar to other ad headline generation methods Xu et al. (2019); Hughes et al. (2019) methods for a fair comparison to Self-Critical MLM.

### Ablations

We trained a model with the same architecture, number of parameters and input as the proposed models but without MLM pre-training and separately without Self-Critical loss to study the impact of the proposals.

We also trained a model with MLM pre-training but fine-tuning only using the primary first product from each campaign instead of using all the products. This is interesting since some of the campaigns are cohesive to a degree with similar products and using only one product improves training time and inference latency.

We also report overlap metrics for model that does not use length normalization and post-processing discussed in equation10. We also include results for model that uses BERT Base as the base model instead of BERT Large.

## 5 Results

### Overlap with Approved Headlines

The first evaluation criterion we adopt is overlap Sharma et al. (2017) of model headlines with subject-matter-experts approved human-submitted headlines from the test set (Table1).

Masking the source product title words reduces the performance as the titles and headlines do not follow the same sentence structure and distribution. Adding product category embedding reduces performance and our hypothesis is that this is because the base model cannot be pre-trained with these embeddings. Only using one title achieves lesser but respectable performance, highlighting the efficacy of multi-product conditioning.

"No pre-training of MLM" highlights the advantage of using non-pretrained Transformer based architecture over bi-LSTM. 'Proposed MLM' shows the advantage of using pre-training, BERT Large and only masking the headline. 'Proposed Self-Critical MLM' achieves the best scores across all the metrics and highlights the applicability of our proposed approach 



### Quality and Grammar Audits

We also conducted large scale crowd-sourced evaluation studies of the headlines with over 150,000 judgments. All headlines are shuffled and each headline is rated by 3 random and double-blind crowd-sourced auditors. The quality is judged on a 3-point scale of [1. Incorrect or Irrelevant, 2. Correct, 3. Correct and Attractive] and we use the mode of the 3 judgments.

In this double-blind audit, the auditors were not aware of the source of the headlines and we were not aware of the identity or demographics of any auditor. More details about the workforce may be found in the platform documentation Ground Truth (2021). In order to determine the compensation for the crowd-sourced workers, we used the guideline provided by the crowd-sourcing platform to "choose a price consistent with the approximate time it takes to complete a task" Visible in the Console while creating the Labeling Labeling (2021) job). We thus first conducted an internal audit by volunteers across our organization to determine the time required to complete the task (average 21.59s) and then used the remuneration recommended for the corresponding time range ($0.12 for 20s - 22s).

Table 2 summarizes the quality audits. The SC-biLSTM model performed worse compared to human-submitted headlines. The proposed SC-MLM model achieves the highest average rating and the most number of perfectly rated headlines. Using just a single product does produce correct headlines with 8% faster inference latency but fails to produce attractive headlines due to lack of input from multiple products.

We also conducted Grammar specific audits (N \(\approx\) 10000) in which the grammar of the headlines is judged independently. 98.13% of SC-MLM and 98.12% of MLM generated headlines were judged to have correct grammar against 93.14% of human submitted headlines.

Table 3 shows a sample of headlines for campaigns in the blind test-set. Excessive keyword stuffing in source product titles does hamper headline quality at times and post-filtering using beam

\begin{table}
\begin{tabular}{l c c c c c} \hline \hline Model & Rouge-L & CIDEr & BLEU-4 & METEOR & Avg. Cos. Sim. \\ \hline _Baseline bi-LSTM Pointer Network model_ & & & & & \\ bi-LSTM & - & - & - & - & - \\ Self Critical bi-LSTM & 0.62 & 0.01 & 1.06 & 0.42 & -4.31 \\ \hline _MLM Baselines and Ablations (Single Product and No Self Critical Training)_ & & & & & \\ First Product Only & 2.14 & 0.19 & 5.03 & 3.55 & 0.36 \\ First Product and Category embedding & 1.52 & 0.13 & 4.18 & 2.938 & 0.15 \\ \hline _Proposed MLM and Ablations (Multiple Products and No Self Critical Training)_ & & & & & \\ Using BERT Base instead of BERT Large & 2.85 & 0.22 & 4.96 & 3.58 & 1.53 \\ No pre-training of MLM (Training from scratch) & 3.38 & 0.27 & 5.72 & 3.79 & -0.04 \\ Additional Source Titles Masking & 4.13 & 0.29 & 4.42 & 5.41 & -2.09 \\
**Proposed MLM** & 5.08 & 0.42 & 7.49 & 5.46 & 1.31 \\ \hline _Proposed Self-Critical MLM (SC-MLM) and Ablation_ & & & & & \\ No beam search normalization and post-processing & 5.37 & 0.43 & 7.81 & 5.61 & 1.96 \\
**Proposed Self-Critical MLM** & **6.33** & **0.55** & **9.11** & **6.14** & **3.75** \\ \hline \hline \end{tabular}
\end{table}
Table 1: Absolute improvement over baseline in terms of overlap measures with over 50,000 manually approved human-submitted headlines from the test set. We have reported the **differences** in the F1 of Rouge-L and BLEU-4 scores to the baseline bi-LSTM model. ‘Avg. Cos. Sim.’ is the average cosine similarity of model headlines to the human-submitted headlines measured using an independently pre-trained Language Model.

\begin{table}
\begin{tabular}{c c c c} \hline \hline  & SC-biLSTM & MLM - Single Product & Proposed MLM & Proposed SC-MLM \\ \hline \multicolumn{4}{l}{\% improvement in mean rating over human-submitted headlines} \\  & -9.87\% & 0.40\% & 1.15\% & **2.07\%** \\ \hline \multicolumn{4}{l}{\% improvement in number of headlines} \\ Rated \(\geq\) 2 out of 3 & -4.99\% & **2.75\%** & 2.42\% & 2.37\% \\ Rated 3 out of 3 & -42.96\% & -0.06\% & 1.22\% & **6.53\%** \\ \hline \hline \end{tabular}
\end{table}
Table 2: Comparison of model-generated headlines to human-submitted headlines on a 3-point scale quality audit of a random blind test set (N \(\approx\) 5000).



search score helps to filter them out.

We do observe cases where both the models generate the same headline. This is an artifact of the fact that both the models share the first 15 epochs. The SC-MLM model generates more descriptive headlines and both models are able to abstract the product qualities.

## 6 Conclusion

Ad headline generation is a difficult problem owing to the varying nature of retail product attributes. A lot of historical methods focus on template based creation of ad headlines that are not very expressive.

We demonstrated a new NLG based method to generate headlines for multiple products. Our method achieves highest score in overlap metrics, quality audits and grammar audits compared to the baselines and human-submitted headlines.

Masked Language Models were relatively unexplored for ad headline generation and we were able to demonstrate their utility. We further extended the performance of the model by using Reinforcement Learning. The method only changes the training procedure without impacting inference latency. Thus, our work contributes to both SOTA and practical business applications.

The approach can also be used for any other NLG task.

\begin{table}
\begin{tabular}{l c c c} \hline \hline
* Retail / Event Supplies & Secure your cash with Royal Cash \\ \hline Blue Copy 5 Anti-Agging Body Left, Pregnancy Swedish Marks Prevention and Removal Cream 5 Ox & Blue Copter 5 Anti-Agging Products & Discover Omnotics Best Stelling Products & Say Goodbye to Stretch Marks \\ \hline Cosy House Collection Twin Size Bed Sheets: Green Reading Set: Deep Pocket
- Extra Soft Luxury... & Soft + Hypoulrecognise Twist Sheets
- These Will Change Your Life. & Lowry Twin Shoots
- Those Will Change Your Life. \\ \hline Canon Deblasual Valuicchio Day Heart Scakers: Ixels + Ishak + Ishak, 2167 & Share the Love with this Classroom Decor & Valentire's Dry Celebrations & Show your Valentine’s Dye \\ \hline Canon G42 DORIS LR Battle, Compatible Up Proxima RGQ20 and (5020) MozaffeThakes Filters & Print more for less with SuperTank & Ali-in-one solution for professional grade prints. & Ali-in-one solution for professional grade prints. \\ \hline LABILUS Home Xu MAX case, (Rugged Armore Sense) TPU Soil Stripes & 366? Protection Heavy Drory for iPhose Xu Max & Rugged Fracture Case for iPhose Xu Max & Rugged Armore Protective Case for iPhose Xu Max \\ \hline Biocon Cook off Banker, Government Gift Basket, Deticious Bitcoin Acridly Document Is Count. & Valentine Girls & Gourrent Checkolate Girl Busides & Gournet Heliday Girl Banides \\ \hline La Applegate Directering Iron Wand with Glove And Z Clips
- 34 to Inclh (18-25mm) Conical & Paperted Cutting Winds with Glove and Z & Le Angiague Teperd Cutting Iron Wand \\ \hline JUNR Brands Loelero Re-SBL London & Headows For Every Adventure & Headows for Evvy Adventure & BBL Hedrahards for Adventure \\ \hline JumpRG Portable Car Battery Jump Starters at50,000nm, GOBA Park, Mini Automotive Power Book.... & Postulate Jump and go Impurature & Jump and Go Poritable Car Jump Startar & Jump and Go Poritable Jump Sumtures \\ \hline decant Silver Metal Thin Edge yS7 Picture Frames, Silver Thin Profile Photo Firms & Life-Style-Living & Metal Thin Edge Picture Frames & Thin Edge Picture Frames \\ \hline CARSYS Coating Thickness Gauge & Coating Thickness Gunge & Coating Thickness Gunge
- Range Precision & Coating Thickness Gunge \\ DPN-61 Esteander Range Precione Probe FPrint Meter for Car. & Coating Thickness Gunge & Coating Thickness Gunge
- Range Precision & Coating Thickness Gunge \\ \hline Rich \& Creamity Butermink Syrup & Fresh and Premium Battermik Syrup & Rich and Crosny Buttermik Syrup
- Take Great \\ Original Flueve by Uncle Book’s White Country & Fresh and Premium Battermik Syrup & Rich and Crosny Buttermik Syrup
- Take Great \\ \hline Sesume Street Exn Fine Tree Funny Hunner Push Add Man Groups T-Shirt & Seams Street Trees for Adults & Seams Street Men’s Shirts & Seams Street Men’s Shirts \\ Apparel Small, Orange & Agree Unbreakable East Trip [ Pick 69] & & \\ \hline Azusey Datebook Enat Trip [ Pick 69] & AVUSE Fast Lightning Charging Code & AGVEE Fast Phone 11 Xúlos 10x XR & AGVEE Heavy Drug iPhose ii1 XuXS \\ \multicolumn{4}{c}{} \\ \multirow{2}{*}{Cake} & Doax MX XX®, Private 10x (10x Ins) & for iPhose & Calile & XXuke
 



## References

* Bahdanau et al. (2017) Dzmitry Bahdanau, Philemon Brakel, Kelvin Xu, Anirudh Goyal, Ryan Lowe, Joelle Pineau, Aaron Courville, and Yoshua Bengio. 2017. An Actor-Critic Algorithm for Sequence Prediction. _arXiv:1607.07086__[cs]_. ArXiv: 1607.07086.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. _arXiv:1810.04805__[cs]_. ArXiv: 1810.04805.
* Dong et al. (2019) Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019. Unified Language Model Pre-training for Natural Language Understanding and Generation. _arXiv:1905.03197__[cs]_. ArXiv: 1905.03197.
* MTurk with Ground Truth (2021) Using MTurk with Ground Truth. 2021. [link].
* Hughes et al. (2019) J. Weston Hughes, Keng-hao Chang, and Ruofei Zhang. 2019. Generating Better Search Engine Text Advertisements with Deep Reinforcement Learning. In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, KDD '19, pages 2269-2277, Anchorage, AK, USA. Association for Computing Machinery.
* Joulin et al. (2016) Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2016. Bag of tricks for efficient text classification. _arXiv preprint arXiv:1607.01759_.
* Kingma and Ba (2014) Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. _International Conference on Learning Representations_.
* Labeling (2021) Ground Truth Labeling. 2021. Create a labeling job.
* Lewis et al. (2019) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. BART: Denoising Sequence-to-Sequence Pretraining for Natural Language Generation, Translation, and Comprehension. _arXiv:1910.13461__[cs, stat]_. ArXiv: 1910.13461.
* Lin (2004) Chin-Yew Lin. 2004. ROUGE: A Package for Automatic Evaluation of Summaries. In _Text Summarization Branches Out_, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
* Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Weijing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
* M. P. Duft and S. R. H. Tetlifford (2019) Sponsored Advertising policies. [link].
* Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. _OpenAI Blog_, 1(8):9.
* Ramachandran et al. (2016) Prajit Ramachandran, Peter J. Liu, and Quoc V. Le. 2016. Unsupervised pretraining for sequence to sequence learning. _arXiv preprint arXiv:1611.02683_.
* Ranzato et al. (2016) Marc'Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. 2016. Sequence Level Training with Recurrent Neural Networks. _arXiv:1511.06732__[cs]_. ArXiv: 1511.06732.
* Rennie et al. (2017) Steven J. Rennie, Etienne Marcheret, Youssef Morueh, Jarret Ross, and Vaibhava Goel. 2017. Self-critical Sequence Training for Image Captioning. _arXiv:1612.00563__[cs]_. ArXiv: 1612.00563.
* Rothe et al. (2019) Sascha Rothe, Shashi Narayan, and Aliaksei Severyn. 2019. Leveraging Pre-trained Checkpoints for Sequence Generation Tasks. _arXiv:1907.12461__[cs]_. ArXiv: 1907.12461.
* See et al. (2017)bigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get To The Point: Summarization with Pointer-Generator Networks. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1073-1083, Vancouver, Canada. Association for Computational Linguistics.
* Sharma et al. (2017)Shikhar Sharma, Layla El Asri, Hannes Schulz, and Jeremie Zumer. 2017. Relevance of Unsupervised Metrics in Task-Oriented Dialogue for Evaluating Natural Language Generation. _arXiv:1706.09799__[cs]_. ArXiv: 1706.09799.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, _Advances in Neural Information Processing Systems 30_, pages 5998-6008. Curran Associates, Inc.
* Williams (1992) Ronald J. Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. _Machine Learning_, 8(3):229-256.
* Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remii Louf, Morgan Funtowicz, and Jamie Brew. 2020. HuggingFace's Transformers: State-of-the-art Natural Language Processing. _arXiv:1910.03771__[cs]_. ArXiv: 1910.03771.
* Wu et al. (2016) Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Khrikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2016. Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation. _arXiv:1609.08144__[cs]_. ArXiv: 1609.08144.



* Xu et al. (2019) Peng Xu, Chien-Sheng Wu, Andrea Madotto, and Pascale Fung. 2019. Clickbait? Sensational Headline Generation with Auto-tuned Reinforcement Learning. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 3065-3075, Hong Kong, China. Association for Computational Linguistics.
* Zhu et al. (2020) Jinhua Zhu, Yingce Xia, Lijun Wu, Di He, Tao Qin, Wengang Zhou, Houqiang Li, and Tie-Yan Liu. 2020. Incorporating BERT into Neural Machine Translation. _arXiv:2002.06823 [cs]_. ArXiv: 2002.06823.