text: 285,131,1488788646966800385,1141006043218108419,Clara Isabel Meister,"Neural language models are really good at explaining held-out data. So when we sample from them, why do they yield dull and degenerate text?  Our paper analyzes this behavior using information theory, and corrects for it with a new sampling principle: <LINK> <LINK> Humans use natural language as a vehicle for communication—exactly the concept info theory studies. Results from psycholinguistics tell us we use it in an efficient and somewhat predictable way. One consequence is that we pack an expected amount of information into each word. Can we decode from a probabilistic language generator in a manner that mimics this process? Yes! Our paper proposes a new decoding principle: Instead of sampling only high probability words, we should instead sample from the set of words whose information content is close to the expected information content, i.e., the model's conditional entropy. We find that our decoding method leads to fewer repetitions than nucleus or top-k sampling while also performing strongly in quality ratings. Joint work with @tpimentelms @ryandcotterell and Gian! <LINK>",https://arxiv.org/abs/2202.00666,"Despite achieving incredibly low perplexities on myriad natural language corpora, today's language models still often underperform when used to generate text. This dichotomy has puzzled the language generation community for the last few years. In this work, we posit that the abstraction of natural language as a communication channel (\`a la Shannon, 1948) can provide new insights into the behaviors of probabilistic language generators, e.g., why high-probability texts can be dull or repetitive. Humans use language as a means of communicating information, and do so in a simultaneously efficient and error-minimizing manner; they choose each word in a string with this (perhaps subconscious) goal in mind. We propose that generation from probabilistic models should mimic this behavior. Rather than always choosing words from the high-probability region of the distribution--which have a low Shannon information content--we sample from the set of words with information content close to the conditional entropy of our model, i.e., close to the expected information content. This decision criterion can be realized through a simple and efficient implementation, which we call typical sampling. Automatic and human evaluations show that, in comparison to nucleus and top-k sampling, typical sampling offers competitive performance in terms of quality while consistently reducing the number of degenerate repetitions. ",Typical Decoding for Natural Language Generation,5,"['Neural language models are really good at explaining held-out data. So when we sample from them, why do they yield dull and degenerate text? \n\nOur paper analyzes this behavior using information theory, and corrects for it with a new sampling principle: <LINK> <LINK>', 'Humans use natural language as a vehicle for communication—exactly the concept info theory studies. Results from psycholinguistics tell us we use it in an efficient and somewhat predictable way. One consequence is that we pack an expected amount of information into each word.', 'Can we decode from a probabilistic language generator in a manner that mimics this process?', ""Yes! Our paper proposes a new decoding principle: Instead of sampling only high probability words, we should instead sample from the set of words whose information content is close to the expected information content, i.e., the model's conditional entropy."", 'We find that our decoding method leads to fewer repetitions than nucleus or top-k sampling while also performing strongly in quality ratings.\n\nJoint work with @tpimentelms @ryandcotterell and Gian! https://t.co/R2XO2usU0g']",22,02,1095