HTLM: Hyper-Text Pre-Training and Prompting of Language Models

1 INTRODUCTION . The vast majority of text used to pretrain language models is extracted from web pages , while discarding any markup they contain Liu et al . ( 2019 ) ; Brown et al . ( 2020 ) ; Raffel et al . ( 2019 ) ; Lewis et al . ( 2019 ) . We argue that this HTML should not be ignored ; it enables new forms of highly effective language model pretraining and prompting with structured document-level supervision . Hyper-text , such as the HTML found in the Common Crawl1 , has a number of advantages for pretraining over plain text . It often encodes high-level properties of different parts of the documents , which are difficult to infer from the text alone . For example , < title > elements can be excellent summaries of the < body > of a document , while element class and id attributes can encode categorical properties of documents . Such supervision is highly diverse , depending on what the website authors choose to present , and provides close proxies for many NLP tasks we aim to later solve . Modeling hyper-text allows us to introduce structured prompting of language models . We design prompts that incorporate the established semantics of HTML to better control for the desired model output . This includes , for example , performing zero-shot summarization by asking the model to infill < title > tags in a web page . And , the fact that we jointly model text and hyper-text formatting also allows for effective auto-prompting . If we have even a few examples for a new task , we can directly ask the model to format them in HTML , and templatize the result to define the new prompt . Our HyperText Language Model ( HTLM ) is trained on 23TB of simplified HTML which we automatically extract from common crawl dumps ( see Section §2.1 ) . We use a modified BART denoising objective Lewis et al . ( 2019 ) that randomly masks spans of hyper-text and aims to reconstruct the original input . We extend the original masking with a new size hint scheme , where each mask is associated with an integer that provides a noisy hint for the size of the masked text , to allow for more fine grained task-specific length priors when prompting the final model ( see Section §2.3 ) . Figure 1 shows an example mask that should be reconstructed with a phrase that contains roughly 12 tokens . 1https : //commoncrawl.org/ Through extensive experiments , we show that our HTLM achieves highly effective transfer for a wide range of end tasks and supervision levels . It matches or exceeds the performance of comparably sized text-only LMs for zero-shot prompting and full fine-tuning on GLUE , while also setting new state-of-the-art performance levels for zero-shot summarization with a gain of up to 8 ROUGE-1 points . It also allows few shot learning for problems that are less easily reduced to text-only inputs , such table to text generation . Following methodology introduced by Le Scao & Rush ( 2021 ) , we further find that hyper-text prompts provide more data efficiency to the HTLM model than plain text prompts do for existing LMs , being effectively equivalent to having up to a thousand extra training examples . Finally , we see that the HTLM model is highly effective at auto-prompting itself , in some cases rivaling the performance of manually engineered prompts . In summary , our contributions include : • We present the first hyper-text language model ( HTLM ) , trained on 23TB of simplified HTML data from the common crawl . • Our new hyper-text prompting scheme uses both the well-established semantics of HTML and new size hints on prompt masks to provide more fine-grained control of new task specifications . • We demonstrate consistently strong transfer from HTLM to a range of tasks at differing supervision levels , including improving the best-known zero-shot summarization numbers by up to 8 ROUGE-1 points . • Following Le Scao & Rush ( 2021 ) , our data efficiency analysis shows that hyper-text prompts are worth more to the HTLM model than plain text prompts are for existing LMs , being effectively equivalent to having up to a thousand extra training examples . • We demonstrate the HTLM directly supports auto prompting for new tasks , by simply asking it to format any available examples in HTML , often rivaling or surpassing previous manually engineered prompts . • We release all code and models to support future HTLM research . 2 HYPERTEXT LANGUAGE MODEL ( HTLM ) HTLM is trained on a large corpus of simplified HTML , which is automatically extracted from the common crawl ( Section §2.1 ) . We use a BART-style denoising autoencoder with span masking ( Section §2.2 ) , extended to allow size hints during reconstruction of the original text ( Section §2.3 ) . 2.1 MINIMAL HTML . Although HTML contains supervision signals to natural language , the majority of HTML in a modern web page does not provide any significant form of supervision for pretraining . For example , a large portion of a webpage is JavaScript code or CSS , which provides more aesthetics to the page rather than document-level information . Coupling this with the challenges of training transformers on very long sequence lengths ( Choromanski et al. , 2020 ; Wang et al. , 2020 ; Beltagy et al. , 2020 ) , it was important to automatically convert web pages to a simplified form , which we call Minimal-HTML ( MHTML ) , as defined below . We remove all sub-trees of the HTML DOM2 which do not contain textual elements of a certain character size ( 128 for standard textual elements , 64 for lists/tables/spans ) . We also filter out all headers , footers , copyrights , forms , and iFrames . We fold consecutive < div > elements into a singular < div > element with merged attributes . We also remove all attributes which are not class or id attributes . Lastly , we skip all MHTML documents whose ratio of text to HTML is not greater than 0.46 . Particularly we noticed that MHTML documents whose ratio of text to HTML is low , the average quality of the document tends to be lower as well . These specific cut-offs were found by visually inspecting a set of Common Crawl ( CC ) documents after application of aforementioned transforms ensuring both a high quality of kept documents while also not filtering too large amount of data . Furthermore we filter out all documents who have a lang attribute that is not set to en . Applying these deterministic transformations removes on average 94 % of characters from a raw webpage while maintaining the general markup of the document . Furthermore , it allowed close to 85 % of MHTML documents to fit into 1024 BPE tokens ; the maximum token length for BART and many other existing language models . One by-product of this type of filtering is that it also produced high-quality documents by default3 ; thus , we opted out of model-based filtering of documents such as CC-100 ( Conneau et al. , 2019 ) . We used the January 2021 snapshot of Common Crawl , which provided us with 23 Terabytes of MHTML text after filtering . 2.2 MODEL . We adopt a BART-style denoising auto-encoder ( Lewis et al. , 2019 ) for several reasons . We want to predict arbitrary substrings within the MHTML , conditioned on the rest of the document . This allows us to equally easily ( 1 ) use masks during prompting to mark where to generate text associated with model outputs within a web page , and ( 2 ) automatically generate prompts by wrapping plain text training examples in masks that allow the model to mark them up by generating MHTML formatting . We also do not know in advance exactly how much text needs to be generated in each case , thereby ruling out the use of more traditional masked language models . We maintain the random masking in the BART objective , not adding anything specific to HTML , as random masking is the super-set of all underlying masking policies . For all of our experiments , we adopt the same architecture as BART-Large and initialized our models with the BART-Large checkpoint . This model has roughly 400 million parameters . We trained our augmented BART model for a total of 330,000 steps on 256 GPUs with an effective batch size of 8192 . We initialize our model with the original BART-Large model . We train using the Adam optimizer ( Kingma & Ba , 2014 ) and a polynomial decay learning rate scheduler with a peak learning rate of 4e−5 and 10 , 000 warm-up steps . We do not use the sentence shuffling from the original BART objective , and select a Poisson λ of 3.5 for sampling span lengths for masking . We set dropout in the attention to 0.1 for the first 170k steps , reducing it to 0.0 thereafter . We also filter out data to only English ( en ) after 170k steps using FastText Joulin et al . ( 2016 ) . We noticed the perplexity plateaued around 170k steps which is why we simplify the learning process by removing dropout and applying stronger filtering of the English language . 2.3 SIZE HINTS . BART allows each mask to be replaced with multiple tokens during the reconstruction . During pre-training , BART masks a span with the length sampled from a Poisson distribution ; thus , the model must learn to implicitly predict the length of the masked text . A fundamental problem we 2The DOM or Document Object Model is an interface that treats an HTML document as a tree structure wherein each node is an object representing a part of the document . 3Much of the noise in existing text collections derived from the common crawl comes from artifacts that are introduced when returning the text in the relatively arbitrary order it appeared in the original HTML , before the markup was stripped . encountered when trying to use standard BART for zero-shot generative prompting is the inability to control the length of the generated text for each mask , even when using various decoding strategies like length penalties . To allow for more control , we augment BART ’ s masking scheme by introducing size hints . Specifically , we tokenize the noisy estimate of the length of a span directly and insert it right after the span mask token . For example , given the correct mask length m , we insert n 〈mask〉 tokens where n is max ( 1 , bN ( m , m ∗ ) c ) and is a hyperparameter representing how noisy we want these size hints to be . By optionally injecting size hints , we can prompt the model to generate text of roughly some specific length , or by not injecting size hints , we allow the model to model the mask size implicitly . We give size-hints to 80 % of masks with the noisiness of size hints = 0.1 . 3 HTML-BASED PROMPTING . We use the HTML-based prompting scheme for a range of generation and classification tasks . Broadly , we use HTML templates–either selected manually or generated by the model itself by auto-prompting–to specify the HTML structure of the task . The template is then instantiated with the task input and placeholder mask tokens for the output . The model uses this instantiated template as a prompt . Because BART models reconstruct the full input , we rely on simple heuristics to match the prefix/suffix around any masks and extract the final output .

Summary: This paper introduces a large-scale LM trained directly on the raw HTML in a large-scale web crawl (the common crawl corpus). The resulting model is able to utilize the structure in HTML documents for a variety of tasks such as zero shot summarization, fine-tuning, classification and more. It looks like using structured data for pre-training and creating prompts provides a variety of advantages in LMs. The paper has a comprehensive set of experiments and a thorough ablation study.