text: 448,92,1415928833488785409,1515424688,Armen Aghajanyan,"I'm excited to announce our new pre-training paper: HTLM: Hyper-Text Pre-Training and Prompting of Language Models (<LINK>) where we unlock new ways of priming and automatically generating prompts by pre-training on simplified HTML. Modeling HTML has a ton of advantages: it is easily gathered at scale, it provides rich end-task like supervision (i.e. id attributes encode category information) and allows for structured prompting that follows semantics of HTML, i.e. zero-shot summarization by infilling &lt;title&gt; We train a BART-like model on over 20T of simplified HTML with stochastic size hints over the masks to allow for more fine-grained control during prompting. We call this model HTLM (Hyper-Text Language Model). Turns out that we can represent a good amount of NLP tasks as HTML prompts. A prototypical example is doing summarization by infilling a &lt;title&gt;. Below we ask HTLM to do exactly this with a hint that our generated mask should be roughly 12 tokens. <LINK> Furthermore, by doing clever masking and asking HTLM to generate the most likely hyper-text formatting for any available training data, we can automatically generate valid prompts from very few examples. <LINK> We measure the zero-shot performance of manually found prompts and auto-generated prompts (with and without size hints) and discover that we’re able to consistently outperform previous SOTA zero-shot summarization. <LINK> Size hints turn out to be quite important when doing manual prompt tuning as traditionally we would have to find a prompt that both communicates the semantics of the task as well as the length of the generated output, whereas size-hints allow you to focus only on the former. <LINK> We also get competitive results on zero-shot classification results. <LINK> Furthermore we are able to do k=1-shot table-to-text generation by directly representing the tables in HTML. We do well on fine-tuning HTLM on these tasks as well. <LINK> We follow up by doing an analysis from Scao and Rush, “How Many Data Points is a Prompt for HTLM and find that HTML prompts are generally worth more to HTLM than NL prompts to other pre-trained models. <LINK> This was work done with amazing co-authors @diametralis @ml_perception @mandarjoshi_ @Hu_Hsu Gargi Ghosh and @LukeZettlemoyer",https://arxiv.org/abs/2107.06955,"We introduce HTLM, a hyper-text language model trained on a large-scale web crawl. Modeling hyper-text has a number of advantages: (1) it is easily gathered at scale, (2) it provides rich document-level and end-task-adjacent supervision (e.g. class and id attributes often encode document category information), and (3) it allows for new structured prompting that follows the established semantics of HTML (e.g. to do zero-shot summarization by infilling title tags for a webpage that contains the input text). We show that pretraining with a BART-style denoising loss directly on simplified HTML provides highly effective transfer for a wide range of end tasks and supervision levels. HTLM matches or exceeds the performance of comparably sized text-only LMs for zero-shot prompting and fine-tuning for classification benchmarks, while also setting new state-of-the-art performance levels for zero-shot summarization. We also find that hyper-text prompts provide more value to HTLM, in terms of data efficiency, than plain text prompts do for existing LMs, and that HTLM is highly effective at auto-prompting itself, by simply generating the most likely hyper-text formatting for any available training data. We will release all code and models to support future HTLM research. ",HTLM: Hyper-Text Pre-Training and Prompting of Language Models,11,"[""I'm excited to announce our new pre-training paper: HTLM: Hyper-Text Pre-Training and Prompting of Language Models (<LINK>) where we unlock new ways of priming and automatically generating prompts by pre-training on simplified HTML."", 'Modeling HTML has a ton of advantages: it is easily gathered at scale, it provides rich end-task like supervision (i.e. id attributes encode category information) and allows for structured prompting that follows semantics of HTML, i.e. zero-shot summarization by infilling &lt;title&gt;', 'We train a BART-like model on over 20T of simplified HTML with stochastic size hints over the masks to allow for more fine-grained control during prompting. We call this model HTLM (Hyper-Text Language Model).', 'Turns out that we can represent a good amount of NLP tasks as HTML prompts. A prototypical example is doing summarization by infilling a &lt;title&gt;. Below we ask HTLM to do exactly this with a hint that our generated mask should be roughly 12 tokens. https://t.co/XC7IPmWdmd', 'Furthermore, by doing clever masking and asking HTLM to generate the most likely hyper-text formatting for any available training data, we can automatically generate valid prompts from very few examples. https://t.co/JebH5Z1kfa', 'We measure the zero-shot performance of manually found prompts and auto-generated prompts (with and without size hints) and discover that we’re able to consistently outperform previous SOTA zero-shot summarization. https://t.co/dRmClKphg6', 'Size hints turn out to be quite important when doing manual prompt tuning as traditionally we would have to find a prompt that both communicates the semantics of the task as well as the length of the generated output, whereas size-hints allow you to focus only on the former. https://t.co/rNQSp4M6xQ', 'We also get competitive results on zero-shot classification results. https://t.co/QImoolZ4DD', 'Furthermore we are able to do k=1-shot table-to-text generation by directly representing the tables in HTML. We do well on fine-tuning HTLM on these tasks as well. https://t.co/pBjtwDgPO4', 'We follow up by doing an analysis from Scao and Rush, “How Many Data Points is a Prompt for HTLM and find that HTML prompts are generally worth more to HTLM than NL prompts to other pre-trained models. https://t.co/JKIzuje5x5', 'This was work done with amazing co-authors @diametralis @ml_perception @mandarjoshi_ @Hu_Hsu Gargi Ghosh and @LukeZettlemoyer']",21,07,2288