text: 19,98,1161443195479371776,280403336,Sean Welleck,"our new paper: ""Neural Text d̶e̶Generation with Unlikelihood Training"" is now on arxiv! (w/ @uralik1, @stephenroller, Emily Dinan, @kchonyc, @jaseweston) <LINK> A step towards solving the case of neural text degeneration 🔎 <LINK> @uralik1 @stephenroller @kchonyc @jaseweston Language models (e.g. GPT-2) tend to produce repetitive and dull ('degenerate') text, especially with greedy or beam search <LINK> We propose 'unlikelihood training', which augments maximum likelihood with penalties on certain candidate tokens <LINK> Using unlikelihood loss at the token and sequence levels reduces repetitions, leading to improved generation quality <LINK>",https://arxiv.org/abs/1908.04319,"Neural text generation is a key tool in natural language applications, but it is well known there are major problems at its core. In particular, standard likelihood training and decoding leads to dull and repetitive outputs. While some post-hoc fixes have been proposed, in particular top-$k$ and nucleus sampling, they do not address the fact that the token-level probabilities predicted by the model are poor. In this paper we show that the likelihood objective itself is at fault, resulting in a model that assigns too much probability to sequences containing repeats and frequent words, unlike those from the human training distribution. We propose a new objective, unlikelihood training, which forces unlikely generations to be assigned lower probability by the model. We show that both token and sequence level unlikelihood training give less repetitive, less dull text while maintaining perplexity, giving superior generations using standard greedy or beam search. According to human evaluations, our approach with standard beam search also outperforms the currently popular decoding methods of nucleus sampling or beam blocking, thus providing a strong alternative to existing techniques. ",Neural Text Generation with Unlikelihood Training,4,"['our new paper:\n\n""Neural Text d̶e̶Generation with Unlikelihood Training""\n\nis now on arxiv! (w/ @uralik1, @stephenroller, Emily Dinan, @kchonyc, @jaseweston) <LINK>\n\nA step towards solving the case of neural text degeneration 🔎 <LINK>', ""@uralik1 @stephenroller @kchonyc @jaseweston Language models (e.g. GPT-2) tend to produce repetitive and dull ('degenerate') text, especially with greedy or beam search https://t.co/EFLQlZdu3a"", ""We propose 'unlikelihood training', which augments maximum likelihood with penalties on certain candidate tokens https://t.co/kxdQYZzjpH"", 'Using unlikelihood loss at the token and sequence levels  reduces repetitions, leading to improved generation quality https://t.co/EeM93TccUd']",19,08,649