text: 251,206,1313816344580808704,1141006043218108419,Clara Isabel Meister,"Beam search is a hack -- we all know it. So, why does it work so damn well? It’s a SOTA algorithm for decoding neural text generators! Our new EMNLP paper presents a framing of beam search that demonstrates it has a cognitive inductive bias. <LINK> <LINK> Formally, we cast beam search as the solution to a regularized decoding problem. Analysis of our “beam search regularizer” reveals a concrete link between beam search and the UID hypothesis from cognitive science. We find that beam search encourages an even distribution of information across the generated text, which is cognitively plausible! In our machine translation experiments, we show that BLEU correlates with an operationalization of the UID hypothesis. This gives us a simple explanation about why beam search works so well with small beam sizes: It enforces a posited cognitive bias from the linguistics literature -- to wit, the UID hypothesis. We also develop a set of novel regularizers, inspired by further work on the UID hypothesis, and decode with them in the regularized decoding framework. Experimentally, we find that our novel regularizers behave in a similar manner to beam search with a small beam size. We conclude the paper arguing that our experimental results give us a plausible, cognitive explanation for beam search’s success as a decoding heuristic for neural text generators, even when the algorithm is far from exact in practice. Joint work with @ryandcotterell and @xtimv.",https://arxiv.org/abs/2010.02650,"Quite surprisingly, exact maximum a posteriori (MAP) decoding of neural language generators frequently leads to low-quality results. Rather, most state-of-the-art results on language generation tasks are attained using beam search despite its overwhelmingly high search error rate. This implies that the MAP objective alone does not express the properties we desire in text, which merits the question: if beam search is the answer, what was the question? We frame beam search as the exact solution to a different decoding objective in order to gain insights into why high probability under a model alone may not indicate adequacy. We find that beam search enforces uniform information density in text, a property motivated by cognitive science. We suggest a set of decoding objectives that explicitly enforce this property and find that exact decoding with these objectives alleviates the problems encountered when decoding poorly calibrated language generation models. Additionally, we analyze the text produced using various decoding strategies and see that, in our neural machine translation experiments, the extent to which this property is adhered to strongly correlates with BLEU. ","If beam search is the answer, what was the question?",6,"['Beam search is a hack -- we all know it. So, why does it work so damn well? It’s a SOTA algorithm for decoding neural text generators! Our new EMNLP paper presents a framing of beam search that demonstrates it has a cognitive inductive bias. <LINK> <LINK>', 'Formally, we cast beam search as the solution to a regularized decoding problem. Analysis of our “beam search regularizer” reveals a concrete link between beam search and the UID hypothesis from cognitive science.', 'We find that beam search encourages an even distribution of information across the generated text, which is cognitively plausible! In our machine translation experiments, we show that BLEU correlates with an operationalization of the UID hypothesis.', 'This gives us a simple explanation about why beam search works so well with small beam sizes: It enforces a posited cognitive bias from the linguistics literature --  to wit, the UID hypothesis.', 'We also develop a set of novel regularizers, inspired by further work on the UID hypothesis, and decode with them in the regularized decoding framework. Experimentally, we find that our novel regularizers behave in a similar manner to beam search with a small beam size.', 'We conclude the paper arguing that our experimental results give us a plausible, cognitive explanation for beam search’s success as a decoding heuristic for neural text generators, even when the algorithm is far from exact in practice. Joint work with @ryandcotterell and @xtimv.']",20,10,1464