Year: 2021

Abstract:
We introduce HTLM, a hyper-text language model trained on a large-scale web crawl. Modeling hyper-text has a number of advantages: (1) it is easily gathered at scale, (2) it provides rich document-level and end-task-adjacent supervision (e.g. class and id attributes often encode document category information), and (3) it allows for new structured prompting that follows the established semantics of HTML (e.g. to do zero-shot summarization by infilling <title> tags for a webpage that contains the input text). We show that pretraining with a BART-style denoising loss directly on simplified HTML provides highly effective transfer for a wide range of end tasks and supervision levels. HTLM matches or exceeds the performance of comparably sized text-only LMs for zero-shot prompting and fine-tuning for classification benchmarks, while also setting new state-of-the-art performance levels for zero-shot summarization. We also find that hyper-text prompts provide more value to HTLM, in terms of data efficiency, than plain text prompts do for existing LMs, and that HTLM is highly effective at auto-prompting itself, by simply generating the most likely hyper-text formatting for any available training data. We will release all code and models to support future HTLM research.

heading: 1 INTRODUCTION
text: The vast majority of text used to pretrain language models is extracted from web pages, while discarding any markup they contain Liu et al. (2019); Brown et al. (2020); Raffel et al. (2019); Lewis et al. (2019). We argue that this HTML should not be ignored; it enables new forms of highly effective language model pretraining and prompting with structured document-level supervision.
Hyper-text, such as the HTML found in the Common Crawl1, has a number of advantages for pretraining over plain text. It often encodes high-level properties of different parts of the documents, which are difficult to infer from the text alone. For example, <title> elements can be excellent summaries of the <body> of a document, while element class and id attributes can encode categorical properties of documents. Such supervision is highly diverse, depending on what the website authors choose to present, and provides close proxies for many NLP tasks we aim to later solve.
Modeling hyper-text allows us to introduce structured prompting of language models. We design prompts that incorporate the established semantics of HTML to better control for the desired model output. This includes, for example, performing zero-shot summarization by asking the model to infill <title> tags in a web page. And, the fact that we jointly model text and hyper-text formatting also allows for effective auto-prompting. If we have even a few examples for a new task, we can directly ask the model to format them in HTML, and templatize the result to define the new prompt.
Our HyperText Language Model (HTLM) is trained on 23TB of simplified HTML which we automatically extract from common crawl dumps (see Section §2.1). We use a modified BART denoising objective Lewis et al. (2019) that randomly masks spans of hyper-text and aims to reconstruct the original input. We extend the original masking with a new size hint scheme, where each mask is associated with an integer that provides a noisy hint for the size of the masked text, to allow for more fine grained task-specific length priors when prompting the final model (see Section §2.3). Figure 1 shows an example mask that should be reconstructed with a phrase that contains roughly 12 tokens.
1https://commoncrawl.org/
Through extensive experiments, we show that our HTLM achieves highly effective transfer for a wide range of end tasks and supervision levels. It matches or exceeds the performance of comparably sized text-only LMs for zero-shot prompting and full fine-tuning on GLUE, while also setting new state-of-the-art performance levels for zero-shot summarization with a gain of up to 8 ROUGE-1 points. It also allows few shot learning for problems that are less easily reduced to text-only inputs, such table to text generation. Following methodology introduced by Le Scao & Rush (2021), we further find that hyper-text prompts provide more data efficiency to the HTLM model than plain text prompts do for existing LMs, being effectively equivalent to having up to a thousand extra training examples. Finally, we see that the HTLM model is highly effective at auto-prompting itself, in some cases rivaling the performance of manually engineered prompts.
In summary, our contributions include:
• We present the first hyper-text language model (HTLM), trained on 23TB of simplified HTML data from the common crawl.
• Our new hyper-text prompting scheme uses both the well-established semantics of HTML and new size hints on prompt masks to provide more fine-grained control of new task specifications.
• We demonstrate consistently strong transfer from HTLM to a range of tasks at differing supervision levels, including improving the best-known zero-shot summarization numbers by up to 8 ROUGE-1 points.
• Following Le Scao & Rush (2021), our data efficiency analysis shows that hyper-text prompts are worth more to the HTLM model than plain text prompts are for existing LMs, being effectively equivalent to having up to a thousand extra training examples.
• We demonstrate the HTLM directly supports auto prompting for new tasks, by simply asking it to format any available examples in HTML, often rivaling or surpassing previous manually engineered prompts.
• We release all code and models to support future HTLM research.
2 HYPERTEXT LANGUAGE MODEL (HTLM)
HTLM is trained on a large corpus of simplified HTML, which is automatically extracted from the common crawl (Section §2.1). We use a BART-style denoising autoencoder with span masking (Section §2.2), extended to allow size hints during reconstruction of the original text (Section §2.3).

heading: 2.1 MINIMAL HTML
text: Although HTML contains supervision signals to natural language, the majority of HTML in a modern web page does not provide any significant form of supervision for pretraining. For example, a large portion of a webpage is JavaScript code or CSS, which provides more aesthetics to the page rather than document-level information. Coupling this with the challenges of training transformers on very long sequence lengths (Choromanski et al., 2020; Wang et al., 2020; Beltagy et al., 2020), it was
important to automatically convert web pages to a simplified form, which we call Minimal-HTML (MHTML), as defined below.
We remove all sub-trees of the HTML DOM2 which do not contain textual elements of a certain character size (128 for standard textual elements, 64 for lists/tables/spans). We also filter out all headers, footers, copyrights, forms, and iFrames. We fold consecutive <div> elements into a singular <div> element with merged attributes. We also remove all attributes which are not class or id attributes. Lastly, we skip all MHTML documents whose ratio of text to HTML is not greater than 0.46. Particularly we noticed that MHTML documents whose ratio of text to HTML is low, the average quality of the document tends to be lower as well. These specific cut-offs were found by visually inspecting a set of Common Crawl (CC) documents after application of aforementioned transforms ensuring both a high quality of kept documents while also not filtering too large amount of data. Furthermore we filter out all documents who have a lang attribute that is not set to en.
Applying these deterministic transformations removes on average 94% of characters from a raw webpage while maintaining the general markup of the document. Furthermore, it allowed close to 85% of MHTML documents to fit into 1024 BPE tokens; the maximum token length for BART and many other existing language models.
One by-product of this type of filtering is that it also produced high-quality documents by default3; thus, we opted out of model-based filtering of documents such as CC-100 (Conneau et al., 2019). We used the January 2021 snapshot of Common Crawl, which provided us with 23 Terabytes of MHTML text after filtering.

heading: 2.2 MODEL
text: We adopt a BART-style denoising auto-encoder (Lewis et al., 2019) for several reasons. We want to predict arbitrary substrings within the MHTML, conditioned on the rest of the document. This allows us to equally easily (1) use masks during prompting to mark where to generate text associated with model outputs within a web page, and (2) automatically generate prompts by wrapping plain text training examples in masks that allow the model to mark them up by generating MHTML formatting. We also do not know in advance exactly how much text needs to be generated in each case, thereby ruling out the use of more traditional masked language models. We maintain the random masking in the BART objective, not adding anything specific to HTML, as random masking is the super-set of all underlying masking policies.
For all of our experiments, we adopt the same architecture as BART-Large and initialized our models with the BART-Large checkpoint. This model has roughly 400 million parameters.
We trained our augmented BART model for a total of 330,000 steps on 256 GPUs with an effective batch size of 8192. We initialize our model with the original BART-Large model. We train using the Adam optimizer (Kingma & Ba, 2014) and a polynomial decay learning rate scheduler with a peak learning rate of 4e−5 and 10, 000 warm-up steps. We do not use the sentence shuffling from the original BART objective, and select a Poisson λ of 3.5 for sampling span lengths for masking. We set dropout in the attention to 0.1 for the first 170k steps, reducing it to 0.0 thereafter. We also filter out data to only English (en) after 170k steps using FastText Joulin et al. (2016). We noticed the perplexity plateaued around 170k steps which is why we simplify the learning process by removing dropout and applying stronger filtering of the English language.

heading: 2.3 SIZE HINTS
text: BART allows each mask to be replaced with multiple tokens during the reconstruction. During pre-training, BART masks a span with the length sampled from a Poisson distribution; thus, the model must learn to implicitly predict the length of the masked text. A fundamental problem we
2The DOM or Document Object Model is an interface that treats an HTML document as a tree structure wherein each node is an object representing a part of the document.
3Much of the noise in existing text collections derived from the common crawl comes from artifacts that are introduced when returning the text in the relatively arbitrary order it appeared in the original HTML, before the markup was stripped.
encountered when trying to use standard BART for zero-shot generative prompting is the inability to control the length of the generated text for each mask, even when using various decoding strategies like length penalties.
To allow for more control, we augment BART’s masking scheme by introducing size hints. Specifically, we tokenize the noisy estimate of the length of a span directly and insert it right after the span mask token. For example, given the correct mask length m, we insert n 〈mask〉 tokens where n is max (1, bN (m,m ∗ )c) and is a hyperparameter representing how noisy we want these size hints to be. By optionally injecting size hints, we can prompt the model to generate text of roughly some specific length, or by not injecting size hints, we allow the model to model the mask size implicitly. We give size-hints to 80% of masks with the noisiness of size hints = 0.1.

heading: 3 HTML-BASED PROMPTING
text: We use the HTML-based prompting scheme for a range of generation and classification tasks. Broadly, we use HTML templates–either selected manually or generated by the model itself by auto-prompting–to specify the HTML structure of the task. The template is then instantiated with the task input and placeholder mask tokens for the output. The model uses this instantiated template as a prompt. Because BART models reconstruct the full input, we rely on simple heuristics to match the prefix/suffix around any masks and extract the final output.

heading: 3.1 GENERATION PROMPTING POLICIES
text: Given that we have optional size hints for masks, a single prompt can generate a wide variety of text; therefore, we discuss multiple policies to select the prompted results. We can decide not to utilize size hints at all and thus remove the need to use any policies, but this comes at the cost of template robustness. Without size hints, a template not only has to express the semantics of the task, but also needs to match the average target length as well; such prompts are brittle and require careful manual design. However, using hints allows us to decouple generation length from the prompt, greatly improving template reuse across related tasks. It is also possible that for a prompt and a specific subset of the data, HTLM will not generate an output from which we can programmatically extract the generated mask; therefore, our policies for size-hints also mitigate this issue.
For every generation task, we first construct a prompt that can generate the correct text semantically, and then we provide size hints equal to the average target of a subset of the training set, s̄. If, for a particular input, we are not able to extract a value, we run HTLM on the same prompt, but with our size hint set to s̄± i s̄, from which we select the output with the lowest perplexity, we continue this process at most five times where i represents the current index of the policy. If we still cannot find a valid generated answer, we fall back on the auto-template described in the next section. In experiments, we denote HTLM-Manual-NS (not sized) as our manually engineered prompt with no size hint, while HTLM-Manual-S uses the policy defined here for all generation benchmarks.

heading: 3.2 AUTO-PROMPTING
text: To avoid manually engineering prompts, we also explore automatic generation of structured prompts. By training on hypertext, HTLM can learn high-level document semantics that we exploit for prompt creation. We generate prompting templates by asking the model to recover document markups. Specifically, we place 〈mask〉 tokens around every independent block of data (e.g. summary/article). We provide an example of auto-prompting for a sample from the Gigaword summarization dataset (Napoles et al., 2012) with the respective masking in Figure 2 . For our generation experiments, we denote HTLM-Auto-NS (not-sized) as the auto-prompt without using size hints, where HTLM-Auto-S uses the size hints based policy described in the previous section.
We found that HTLM auto-prompting was less effective for classification tasks. We hypothesize that this is because generative targets carry significantly more information than a simple binary target token.

heading: 4 ZERO/ONE-SHOT PROMPTING
text: Perez et al. (2021) argue that zero/few-shot learning cannot happen when prompts are created by tuning on a large amount of development data. To mitigate for this issue all the manual prompts used throughout our experiments are either derived from related papers or developed using a maximum of fifty samples from the train set.

heading: 4.1 GENERATION
text: We evaluate HTLM on summarization, a prototypical generation task. For all summarization benchmarks, we use ROUGE-1/2/L as our primary metrics to stay consistent with other literature (Lin, 2004).
Furthermore we benchmark HTLM on a set of three standard natural language generation tasks. We utilize the official benchmarking scripts provided which report BLEU (Papineni et al., 2002), NIST (Belz & Reiter, 2006), METEOR (Lavie & Agarwal, 2007), ROUGE-L (Lin, 2004), CIDEr (Vedantam et al., 2015) and TER (Snover et al., 2005). We use Li & Liang (2021) for our baselines, and present prefix tuning results with 0.1% of parameters as well.
Gigaword consists of headlines from news articles (Napoles et al., 2012). The target summaries are relatively short, consisting roughly on average of 10 BPE tokens. CNN/Dailymail (Hermann et al., 2015) provides multi-sentence target summaries close to 3 sentences, or roughly 50 tokens. Reddit TIFU (Kim et al., 2018) contains summaries of Reddit posts. Specifically, we use the short subset of data . Compared to our other summarization datasets, this dataset is highly abstractive and not based on news articles. XSum (Narayan et al., 2018) provides abstractive single sentence summaries of news articles. E2E Novikova et al. (2017) is a table-to-text generation dataset containing approximately 50K samples with 8 unique fields from the restaurants domain. WebNLG Gardent et al. (2017) is also a structured generation dataset containing 15 different domains from DBPedia. We report numbers on the Seen (S), Unseen (U) and All (A) subsets of the data. DART Nan et al. (2020) is a open-domain structured generation dataset containing Wikipedia tables.
We manually searched for prompts for each of these datasets using a maximum of 50 data points from the train set to evaluate the prompts. We compare 3 different types of prompts; HTLM-Manual denotes manually engineered prompts with size hints, while HTLM-Auto-S and HTLM-Auto-NS indicate autoprompting with and without size hints respectively. For our baseline, we compare against
PEGASUS (Zhang et al., 2019), the current state of the art for zero shot summarization. PEGASUS was explicitly pre-trained for summarization by masking and generating salient gap sentences from news articles. We present our results in Table 1.
HTLM with manual prompts (HTLM-Manual) and size hints substantially improves over state-of-theart zero-shot summarization results on all four datasets without any tailored pretraining. In particular, we see large improvements of more than 8 ROUGE-L F1 for the Gigaword dataset. Furthermore, size hints-based auto-prompting (HTLM-Auto-S) outperforms PEGASUS in three out of four datasets. Specifically, for the Gigaword dataset, we outperform previous state-of-the-art zero-shot results from PEGASUS by roughly 6 ROUGE-L points. HTLM improvements stem from the fact that HTML-based prompting allows us better control over dataset-specific attributes such as length and style.
For NLG tasks, we required the use of a single training example to get prompting to work sufficiently. We report these one-shot numbers in Table 2. Because these tasks require structured tabular inputs, it is not obvious how to prompt any other text-based pre-trained models. We report other non-trainable baselines such as the grammar based pipeline approaches (TILB/UIT-VNU) in Gardent et al. (2017). To the best of our knowledge, these are the first one-shot table to text, natural language generation results.

heading: 4.2 CLASSIFICATION
text: For prompting in the classification setting, we select 4 datasets to work with. Instead of relying on generative prompting to generate target token(s) denoting the correct class, we instead rely on perplexity measures over the set of all targets to select the correct class. In other words, we select the class for which the perplexity of the corresponding instantiated template is the smallest.
RTE (Bentivogli et al., 2009) is a textual entailment task formulated as binary classification. We place the candidate in a <div> element with the class attribute set to candidate and do the same with the respective hypothesis. In the third element, we utilize the prompt from Brown et al. (2020) with the class attribute set to answer.
BoolQ (Clark et al., 2019) is a yes/no question answering task, also formulated as binary classification for question, passage, and answer triplets. We represent the question as a <div> element with the
itemprop set to https://schema.org/Question, passage as a div element with class attribute passage and answer as a div element with the itemprop set to https://schema.org/Answer. Winogrande (Levesque et al., 2012) consists of adversarially collected Winograd Schema Challenge Levesque et al. (2011) data. We utilize the same template as GPT-3 but place it in a QA style template similar to BoolQ. Please refer to the Appendix for exact templates. HellaSwag The last dataset we evaluate is the commonsense natural language inference task HellaSwag which, due to its adversarial nature, is considered complex (Zellers et al., 2019).
We present our results on zero-shot classification in Table 3. HTLM prompting of classification datasets outperforms the most comparable (in terms of number of parameters) GPT-3 Medium sized model on the majority of tasks, while approaching—and on RTE outperforming—the GPT-3 Large model which consists of roughly double the amount of parameters as HTLM.

heading: 5 FINE-TUNING EXPERIMENTS
text: In addition to our previous prompting results, we also aim to show that HTLM learned representations are useful in the full finetuning setting at least as well as other pre-trained representations. We compare against other pre-training MLM models such as RoBERTa Liu et al. (2019), original BART Lewis et al. (2019), and T5 Raffel et al. (2019) by finetuning on the GLUE benchmark (Wang et al., 2018).
During finetuning, instead of a simple concatenation of sentences from the train set, we place the examples into prompts derived from Le Scao & Rush (2021). We defer to the Appendix for the exact prompts. Given the recent advancements in finetuning, we also report results using the recently proposed R3F method for finetuning (Aghajanyan et al., 2020a) for both RoBERTa and HTLM. We note that there are other numerous pre-training methods we do not compare against (such as Devlin et al. (2018); Clark et al. (2020); He et al. (2020); Meng et al. (2021)) but the goal of our fine-tuning results is to show that there is no significant degradation in quality from training on structured documents.
We present our results in Table 4. Overall HTLM is competitive with other pre-training methods. We also note that we can improve fine-tuning performance by placing the examples into prompts and fine-tuning the classification head. The improvements that we see in terms of prompting have no
adverse effects on fine-tuning but are rather positive, providing further evidence that the proposed approach of structured pre-training is a viable alternative to other methods of pre-training even for fine-tuning.
We also show our fine-tuning results for the table-to-text generation datasets in Table 2. Similar to GLUE fine-tuning, we place all NLG samples into a prompt while fine-tuning. HTLM fine-tuned is able to outperform both variants of the GPT-2 model consistently.

heading: 6 PROMPT DATA EFFICIENCY
text: What does the HTML-based pretraining and prompting scheme offer over one based on the plain text? Le Scao & Rush (2021) explored quantifying how many data points a single prompt was worth. Specifically, they analyzed three different task-specific settings given a pattern (the structure that the inputs are put into) and verbalizer (i.e., yes/no answer to pattern): (1) fine-tuning a classification head (H), (2) fine-tuning the verbalizer of a prompt encoding the semantics of the task (P ), and (3) fine-tuning the prompt but with a verbalizer that is non-sensical (N ).
By carefully selecting the number of data points to be used during training in each setting while matching the end fine-tuning performance, we can empirically measure the efficacy of prompts in terms of data points. We provide the same analysis extended to BART, T5-Large, and HTLM using the same PET prompts provided in Schick & Schütze (2020). For HTLM, we wrap all PET prompts in an HTML element. We select the same datasets that were used in the original paper for our experimentation; MNLI (Williams et al., 2018), BoolQ (Clark et al., 2019), CB De Marneffe et al. (2019), RTE (Bentivogli et al., 2009), WiC Pilehvar & Camacho-Collados (2019).
We first look at the average advantage of fine-tuning a prompt (P ) against a classification head (H) in Table 5. We see that across the board, HTLM prompts—i.e., hypertext prompts applied to HTLM—are worth more than natural language prompts to various other pre-trained models. Compared to RoBERTa-Large on smaller datasets, HTLM’s advantage is close to triple on CB and double on RTE. Furthermore, on WiC, HTLM is the only pre-trained model capable of having a positive training advantage when using prompts. We view this as additional evidence to the benefit of pre-training on structured data on the prompting of a pre-trained model.
We also compare the average advantage of fine-tuning a prompt with a verbalizer (P ) that makes sense against against finetuning a prompt where we change the verbalizer to a random first name (N ). This is important to capture whether the benefits arise from representing the data in their respective
patterns or the coupling of the pattern and the verbalizer. We present our results in Table 6. Relative to the previous P vs. H setting we lose a large amount of advantage, as was similarly seen in (Le Scao & Rush, 2021). Interestingly enough for small datasets such as CB, all of the training advantage of the prompt comes from the pattern in HTLM.
We view this as further evidence that a structured, document level approach to both pre-training and prompting can be seen as a viable alternative to a purely natural language approach.

heading: 7 RELATED WORK
text: GPT-2 Radford et al. (2019) showed that large language models show varying levels of zero-shot performance across NLP tasks when compared to supervised baselines (e.g., rudimentary performance on summarization, but more competitive results on reading comprehension). Brown et al. (2020) through their GPT3 model showed that by further scaling up language models on a large subset of the internet, prompting could be a viable alternative to standard finetuning. The success of GPT3 was largely attributed to massive size and compute-intensive pretraining. By reformulating NLP tasks as cloze-style questions, Schick & Schütze (2020) shows that the prompting capabilities exhibited by GPT3 can occur in language models of a much smaller scale when gradient-based finetuning is combined with task-specific unlabeled data. Follow-up work Tam et al. (2021) improves upon these results without depending on unlabeled data. Unlike GPT-3 and other models which use conventional natural language text-based prompting, we focus on a new hyper-text based prompting scheme using generative masked language models pre-trained directly over HTML.
For task-specific zero-shot performance, custom pre-training and data augmentation schemes have been developed. For example, PEGASUS (Zhang et al., 2019) proposes a novel pre-training scheme tailored for summarization which involves masking and generating salient gap sentences from a large news corpus. While PEGASUS is capable of doing zero-shot summarization, it offers little control over summary attributes such as length and style which vary across different summarization datasets. WikiTransfer Fabbri et al. (2021) fine-tunes pretrained models on pseudo-summaries, produced from generic Wikipedia data, which contain characteristics of the target dataset, such as the length and level of abstraction. Our proposed model allows fine-grained control over the length of the generated text by specifying the size of the mask. Moreover, by using different prompts, HTLM can produce stylistically varied summaries without dataset-specific augmentation and finetuning.
Another line of work has been looking at a hybrid form of prompting that attempts to optimize very few parameters to solve an end task. For example Li & Liang (2021) argue that optimizing in the continuous prompt space is an effective solution to prompt search while Aghajanyan et al. (2020b) optimize for a low-rank projection of the full parameter space. For simplicity, we only focus on either full-finetuning or zero-shot prompting in this paper.
Attempts have been made to encode architectural priors for structured inputs into transformers as well. Specifically, Ainslie et al. (2020) discuss a new type of model which allows for scalability in input length as well as the ability to encode the structure of the input. We opt to allow HTLM to learn the structure that is available in the HTML directly without encoding any structural priors into the model itself.

heading: 8 CONCLUSION
text: In this paper, we proposed HTLM, a hyper-text language model trained on simplified HTML documents from a large-scale web crawl. We showed that by directly modeling HTML through a BART-like objective, we could do structured zero-shot prompting by representing tasks in HTML. Specifically, we outperform the previous best results on zero-shot prompting for summarization by a wide margin by creating prompts that capture the underlying semantics of each summarization dataset. Furthermore, we show that pre-training on structured data improved full finetuning performance relative to other pre-trained models that only modeled natural language.
We also showed additional advantages of modeling hyper-text, beyond improved accuracy. HTLM can be used for auto-prompt by simply asking the model to recover the document structure from training samples; these auto-prompts on datasets like Gigaword and CNN/DM outperformed previous state-of-the-art zero-shot approaches. Lastly, we provided an in-depth comparison of the training
advantage, in terms of data efficiency, that HTLM had compared to other pre-training approaches. Across the board, HTML prompts were worth more to HTLM than natural language prompts were worth to our baselines, further showing the efficacy of pre-training structured data.
Future work can focus on the scaling laws of structured pre-training and prompting. As was seen from GPT-3, the size of the model and the amount of compute utilized and significant impact on prompting performance.

heading: A APPENDIX
text: A.1 FINETUNING HYPER-PARAMETERS
For our GLUE related experiments the following parameters are used.

Review summaries:
This paper introduces HTLM which is a language model pretrained on a large-scale web crawl hyper-text data. There are several contributions in the paper: * A preprocessing step to filter out noisy components in the web pages is proposed. The resulting simplified format, Minimal-HTML (MHTML), is likely to be composed of high-quality documents which can be used for pretraining. * A modified BART pretraining objective is proposed to inject noisy size hints to control the length of the span to be generated by the model during training. * A new prompting method in the form of HTML templates is described to accomplish generation (e.g., summarization & table-to-text) and classification (e.g., GLUE) tasks. * The resulting pretrained HTLM model has superior performance on zero-shot summarization and can do better than some existing language models pretrained on plain texts.

The paper proposes a new large language model, called HTLM, short for Hyper-Text Language Model. The language model has been trained on 25TB of HTML text. The authors argue that HTML tags provide valuable information regarding document-level structure. Besides the scale, the authors also adopt an optional pre-training strategy where it is possible to provide size hints to govern the length of the generated text or each mask (alongside the BART-like training objective). The trained language model achieves state-of-the-art performance on zero-shot summarization by prompting the model to predict the text within the <title> tags. The method also achieves state-of-the-art performance on several other benchmark NLP tasks, performing better than large pre-trained models that were only trained on text data (as opposed to hyper-text data). The authors also discuss manual and automated prompt construction, and their evaluation shows that the model can also perform better than competitive baselines on the multiple classification tasks. A follow-up analysis also shows that the per-prompt-efficacy is higher for HTLM than for text-only language models. 

This paper introduces a large-scale LM trained directly on the raw HTML in a large-scale web crawl (the common crawl corpus). The resulting model is able to utilize the structure in HTML documents for a variety of tasks such as zero shot summarization, fine-tuning, classification and more. It looks like using structured data for pre-training and creating prompts provides a variety of advantages in LMs. The paper has a comprehensive set of experiments and a thorough ablation study.

This paper presents HTML-based large scale language model pretraining. The model architecture and pre-training method are both based on BART. The author presents the resulting model, HTLM, as a strong zero-shot learner while being on par with other large transformer language models on the GLUE fine tune tasks. the authors show that via various benchmarks in their experiment section.