HTLM: Hyper-Text Pre-Training and Prompting of Language Models
Authors: Armen Aghajanyan
Dmytro Okhonko
Mike Lewis
Mandar Joshi
Hu Xu
Gargi Ghosh
Luke Zettlemoyer
Venue: ICLR 2022 Poster (2022)
Keywords: prompting
nlp
representational learning
priming

Abstract: We introduce HTLM, a hyper-text language model trained on a large-scale web crawl. Modeling hyper-text has a number of advantages: (1) it is easily gathered at scale, (2) it provides rich document-level and end-task-adjacent supervision (e.g. 'class' and 'id' attributes often encode document category information), and (3) it allows for new structured prompting that follows the established semantics of HTML (e.g. to do zero-shot summarization by infilling '<title>' tags for a webpage that contains the input text).  We show that pretraining with a BART-style denoising loss directly on simplified HTML provides highly effective transfer for a wide range of end tasks and supervision levels. HTLM matches or exceeds the performance of comparably sized text-only LMs for zero-shot prompting and fine-tuning for classification benchmarks, while also setting new state-of-the-art performance levels for zero-shot summarization. We also find that hyper-text prompts provide more value to HTLM, in terms of data efficiency, than plain text prompts do for existing LMs, and that HTLM is highly effective at auto-prompting itself, by simply generating the most likely hyper-text formatting for any available training data. We will release all code and models to support future HTLM research. 

Under review as a conference paper at ICLR 2021
HTLM: HYPER-TEXT PRE-TRAINING AND PROMPT- ING OF LANGUAGE MODELS
Anonymous authors Paper under double-blind review
ABSTRACT
We introduce HTLM, a hyper-text language model trained on a large-scale web crawl. Modeling hyper-text has number of advantages: (1) it is easily gathered at scale, (2) it provides rich document-level and end-task tajs-adjaceent supervision (e.g. class and id attributes often encode document category information), and (3) allows for new structured prompting that follows the established semantics of HTML (e.g. do zero-shot summarization by infilling <title> tags for webpage that contains the input text), We show that pretraining with a BART-style denoising loss directly on simplified HTML provides highly effective transfer for wide range of end tasks and supervision levels. HTLM matches or exceeds the performance of comparably sized text-only LMs for zero-shot prompting and fine-tuning for classification benchmarks, while also setting new state-of-the-art performance levels for zero-shot summarization. We also find that hyper-text prompts provide more value to HTLM, in terms of data efficiency, than plain text prompts do for existing LMs, and that HTLM is highly effective at auto-pra suttoprompting itself, by simply generating the most likely hyper-text formatting for any available training data. We will release all code and models to support future HTLM research.
INTRODUCTION
The vast majority of text used to pretrain language models is extracted from web pages, while discarding any markup they contain Liu et al. :0019):Brown et al. (2020); Raffel et al. Lewis al. 2019). We argue that this HTML should not be ignored; it enables new forms of highly effective language model pretraining and prompting with structured document-level supervision.
Hyper-text, such as the HTML found in the Common Craw | has number of advantages for pretraining over plain text. It often encodes high-level properties of different parts of the documents, which are difficult to infer from the text alone. For example, <titles le> elements can be excellent summaries of the <body> of a document, while element class and id attributes can encode categorical properties of documents. Such supervision is highly diverse, depending what the website authors choose to present, and provides close proxies for many NLP tasks we aim to later solve.
Modeling hyper-text allows us to introduce structured prompling of language models. We design prompts that incorporate the established semantics of HTML to better control for the desired model output. This includes, for example. performing zero-shot summarization by asking the model to infill <title> tags in web page. And, the fact that we jointly model text and hyper-text formatting also allows for effective auto prompting. If we have even few examples for new task, we can directly ask the model to format them in HTML, and templatize the result to define the new prompt.
Our HyperText Language Model (HTLM) is trained on 23TB of simplified HTML which we auto- matically extract from common crawl dumps (see Section 1. We use modified BART denoising objective Lewis et al. (2019) that randomly masks spans of hyper-text and aims to reconstruct the original input. We extend the original masking with a new size hint scheme, where each mask is associated with an integer that provides noisy hint for the size of the masked text, to allow for more fine grained task-specific length priors when prompting the final model (see Section $2.3). Figure [ shows example mask that should be reconstructed with phrase that contains roughly 12 tokens.
https: http://ccooooccccccccoo .org/

---

Under review as a conference paper at ICLR 2021
<!DOCTYPE html>
<html
<title> <nask=12 </title>
<body>
<IDOCTIPE
souch korea on nonday announced
<htal>
sxeeрing tax reCorns
<title>
South Rorea Announces Tax
including
corporate Lax cuts boost grouth by atimulating alugeiah: private consumption and business investaent...
Reforns Besnonic Gronth
</title>
<body>
south koraa on monday announced swoeping
</body>
</body>
</btal>
c/html>
Figure 1: An example structured prompt for a simple summarization task, where we ask a generative masked language model generate mask representing the title with average tokens size of 12.
Through extensive experiments, show that HTLM achieves highly effective transfer for wide range of end tasks and supervision levels. It matches or exceeds the performance of comparably sized text-only LMs for zero-shot prompting and full fine-tuning on GLUE, while also setting new state-of-the-art performance levels for zero-shot summarization with gain of up to 8 ROUGE-1 points. It also allows few shot learning for problems that are less easily reduced text-only inputs, such table to text generation. Following methodology introduced by Le Scao & Rush (2021), we further find that hyper-text prompts provide more data efficiency to the HTLM model than plain text prompts do for existing LMs, being effectively equivalent to having up to a thousand extra training examples. Finally, we see that the HTLM model is highly effective at auto tutto-rromping itself, some cases rivaling the performance of manually engineered prompts.
In summary, our contributions include:
We present the first hyper-text language model (HTLM), trained on 23TB of simplified HTML data from the common crawl.
Our new hyper-text prompting scheme uses both the well-established semantics of HTML and new size hints on prompt masks to provide more fine-grained control of new task specifications.
We demonstrate consistently strong transfer from HTLM to a range of tasks at differing supervision levels, including improving the best-known zero-shot summarization numbers by up to 8 ROUGE-1 points.
Following Le Scao & Rush (202)), our data efficiency analysis shows that hyper-text prompts worth more the HTLM model than plain text prompts are for existing LMs, being effectively equivalent to having up to a thousand extra training examples.
We demonstrate the HTLM directly supports auto prompting for new tasks, by simply asking it to format any available examples in HTML, often rivaling or surpassing previous manually engineered prompts.
We release all code and models support future HTLM research.
HYPERTEXT LANGUAGE MODEL (HTLM)
HTLM is trained on a large corpus of simplified HTML,, which is automatically extracted from the common crawl (Section I2.I.. We use a BART-style denoising autoencoder with span masking (Section extended to allow size hints during reconstruction of the original text (Section 22.3)
2.1 MINIMAL HTML
Although HTML contains supervision signals to natural language, the majority of HTML in modern web page does not provide any significant form of supervision for pretraining. For example, large portion of a webpage is JavaScript code CSS, which provides more aesthetics to the page rather than document-level information. Coupling this with the challenges of training transformers on very long sequence lengths (Choromanski al. 2020
Wang al. 2020) Beltagy et al. 2020). was
2

---

Under review as a conference paper at ICLR 2021
important to automatically convert web pages simplified form, which we call Minimal-HT HTML (MHTML), as defined below.
We remove all sub-trees of the HTML DOM D which do not contain textual elements of certain character size (128 for standard textual elements, 64 for lists/tables/spans).. We also filter out all headers, footers, copyrights, forms, and iFrames. We fold consecutive <div> elements into singular <div> element with merged attributes. We also remove all attributes which are not class or id attributes. Lastly, we skip all MHTML documents whose ratio of text to HTML is not greater than 0.46. Particularly we noticed that MHTML documents whose ratio of text to HTML is low, the average quality of the document tends be lower as well. These specific cut-offs were found by visually inspecting a set of Common Crawl (CC) documents after application of aforementioned transforms ensuring both a high quality kept documents while also not filtering too large amount of data. Furthermore we filter out all documents who have a lang attribute that is not set to en.
Applying these deterministic transformations removes on average 94% of characters from a raw webpage while maintaining the general markup of the document, Furthermore, it allowed close to 85% of MHTML documents to fit into 1024 BPE tokens; the maximum token length for BART and many other existing language models.
One by-product of this type of filtering is that it also produced high-quality documents by defaul thus, we opted out of model-based filtering of documents such CC-100 (Conneau al. (019). We used the January 2021 snapshot Common Crawl, which provided us with 23 Terabytes of MHTML text after filtering.
2.2 MODEL
We adopt a BART-style denoising auto-encoder (Lewis et al., 2019) for several reasons. We want to predict arbitrary substrings within the MHTML, conditioned on the rest of the document. This allows us equally easily (1) use masks during prompting mark where to generate text associated with model outputs within a web page, and (2) automatically generate prompts by wrapping plain text training examples in masks that allow the model to mark them up by generating MHTML formatting. We also do not know in advance exactly how much text needs to be generated in each case, thereby ruling out the use of more traditional masked language models. We maintain the random masking in the BART objective, not adding anything specific to HTML, as random masking is the super-set of all underlying masking policies.
For all of our experiments, we adopt the same architecture as BART-Large and initialized our models with the BART-Large checkpoint. This model has roughly 400 million parameters.
We trained our augmented BART model for total of 330,000 steps on 256 GPUs with an effective batch size of 8192. We initialize our model with the original BART-Large model. We train using the Adam optimizer (Kingma & Ba 2014) and polynomial decay learning rate scheduler with a peak learning rate of 4e and 10, 000 warm-up steps.
We do not use the sentence shuffling from the original BART objective, and select Poisson À of 3.5 for sampling span lengths for masking. We set dropout in the attention to 0.1 for the first 170k steps, reducing it to 0.0 thereafter. We also filter out data to only English (en) after 170k steps using FastText Joulin et al. (2016). We noticed the perplexity plateaued around 170k steps which is why simplify the learning process by removing dropout and applying stronger filtering of the English language.
2.3 SIZE HINTS
BART allows each mask to be replaced with multiple tokens during the reconstruction. During pre-training, BART masks span with the length sampled from a Poisson distribution; thus, the model must learn to implicitly predict the length of the masked text. A fundamental problem we
2 The DOM or Document Object Model is interface that treats HTML document tree structure wherein each node is an object representing part of the document.
MMuch of the noise in existing text collections derived from the common crawl comes from artifacts that are introduced when returning the text in the relatively arbitrary order it appeared in the original HTMI,, before the markup was stripped.
3

---

Under review as a conference paper at ICLR 2021
encountered when trying to standard BART for zero-shot generative prompting is the inability to control the length the generated text for each mask, even when using various decoding strategies like length penalties.
To allow for more control, we augment BART's masking scheme by introducing size hints. Specifi- cally, we tokenize the noisy estimate of the length of span directly and insert it right after the span mask token. For example, given the correct mask length m, we insert n (mask) tokens where n is max (1, (M,,, m * e)]) and is hyperparameter representing how noisy we want these size hints to be. By optionally injecting size hints, we can prompt the model to generate text of roughly some specific length, or by not injecting size hints, we allow the model model the mask size implicitly. We give size-hints to 80% of masks with the noisiness of size hints e = 0.1.
HTML- PROMPTING
We use the HTML-based prompting scheme for a range of generation and classification tasks. Broadly, we use HTML complates-either selected manually generated by the model itself by auto-prompting-- to specify the HTML structure of the task. The template is then instantiated with the task input and placeholder mask tokens for the output. The model uses this instantiated template as prompt. Because BART models reconstruct the full input, we rely on simple heuristics to match the prefix/suffix around any masks and extract the final output.
3.1 GENERATION PROMPTING POLICIES
Given that we have optional size hints for masks, single prompt can generate wide variety of text; therefore, we discuss multiple policies to select the prompted results. We can decide not to utilize size hints at all and thus remove the need to use any policies, but this comes the cost of template robustness. Without size hints, template not only has to express the semantics of the task, but also needs to match the average target length as well; such prompts are brittle and require careful manual design. However, using hints allows to decouple generation length from the prompt, greatly improving template reuse across related tasks. also possible that for a prompt and specific subset of the data, HTLM will not generate output from which can programmatically extract the generated mask; therefore, our policies for size-hints also mitigate this issue.
For every generation task, first construct prompt that generate the correct text semantically, and then we provide size hints equal to the average target of subset of the training set, 8. If, for particular input, we are not able extract value, our size hint set to 5 + ies, from which we select the output with the lowest perplexity, we continue this process at most five times where represents the current index of the policy. we still cannot find a valid generated answer, we fall back on the auto-template described in the next section. In experiments, we denote MA11M--Mauul--- (not sized) as our manually engineered prompt with no size hint, while HT M-LMMMaa uses the policy defined here for all generation benchmarks.
HTLM on the same prompt, but with
3.2 PROM-PRGG
To avoid manually engineering prompts, we also explore automatic generation of structured prompts. By training on hypertext, HTLM can learn high-level document semantics that we exploit for prompt creation. We generate prompting templates by asking the model to recover document markups. Specifically, we place (mask) tokens around every independent block of data (c.g. summary/article).
We provide example of auto-prompting for sample from the Gigaword summarization dataset (Napoles et al.) 2012) with the respective masking in Figure 2. For our generation ex- periments, denote HTLM-Auto-NS (not-sized) the auto-prompt without using size hints, where HTLM-Auto-S uses the size hints based policy described in the previous section.
We found that HTLM promppinn less effective for classification tasks. We hypothesize that this is because generative targets carry significantly more information than a simple binary target token.
4

---

Under review as a conference paper at ICLR 2021
<html lang= "on" AI langa GE
<head>
<title>
20 seracts charqes againat
its anbaxsater in bol bolivia
The Rashington Fost
<nack>
</title>
rejects charges against in bolivia
</head>
<body>
<nack>
ETIM
<dly class "post-body
<nack>
department wednesday had received no tornol word from. bolivis that it va5
<pp
atace depaztment seid wednestlay it had received no Tosnal word bolivia
</pp
</div>
</body>
Figure 2: An example of auto-prompting using a sample from the train-set of the Gigaword dataset. HTLM places the summary inside of <title> inside of <head> element, while placing the article <div> element with an entry-contenn content attribute value for attribute class which common news web-sites.
ZERO/ONE-SHOT PROMPTING
Perez et al.
2021 argue that zero/few-shot learning cannot happen when prompts are created by tuning on a large amount of development data. To mitigate for this issue all the manual prompts used throughout our experiments are either derived from related papers or developed using maximum of fifty samples from the train set.
4.1 GENERATION
We evaluate HTLM on summarization, a prototypical generation task. For summarization bench- marks, use ROUGE-1/2LL as primary metrics to stay consistent with other literature (Lin 2004).
Furthermore we benchmark HTLM set of three standard natural language generation tasks. We utilize the official benchmarking scripts provided which report BLEU (Papineni et al. 2002).
NIST (Belz & Reiter 2006). METEOR (Lavie & Agarwal, 2007), ROUGE-L (Lin 2004). CIDEr (Vedantam et al.) 2015) and TER (Snover et al. 2005).. We use Li & Liang 202 for our baselines, and present prefix tuning results with 0.1% of parameters well.
Gigaword consists of headlines from news articles (Napoles et al. 2012). The target summaries are relatively short, consisting roughly on average of 10 BPE tokens.
CNN/Dailymail (Hermann et al. 2015 provides multi-sentence target summaries close sen- tences, or roughly 50 tokens.
Reddit TIFU (Kim et al. 2018) contains summaries of Reddit posts. Specifically, we use the short subset of data. Compared to our other summarization datasets, this dataset is highly abstractive and not based on news articles.
2018 provides abstractive single sentence summaries of news articles. XSum (Narayan et al.
(017) is table-to-text generation dataset containing approximately 50K samples with 8 unique fields from the restaurants domain.
E2E Novikova al.
WebNLG Gardent et al. (2017) is also a structured generation dataset containing 15 different domains from DBPedia. We report numbers on the Seen (S), Unseen (U) and All (A) subsets of the data.
DART Nan et al.
2020 is open-domain structured generation dataset containing Wikipedia tables.
We manually searched for prompts for each of these datasets using a maximum of 50 data points from the train set to evaluate the prompts. We compare 3 different types of prompts; HTLM-Manual denotes manually engineered prompts with size hints, while HTLM-Auto-S and HTLM-Auto- indicate autoprompting with and without size hints respectively. For our baseline, compare against.
5

---

Under review as a conference paper at ICLR 2021
PEGASUS (Zhang et al. 2019). the current state of the art for shot summarization. PEGASUS was explicitly pre-trained for summarization by masking and generating salient gap sentences from news articles. We present our results in Table [
Model
Gigaword
CNN/DM
Reddit TIFU
XSum
PEGASUS-OS
23.39/07.59/20.20
32.90/13.28/29.38
14.66/3.06/10.11
19.27/3.00/12.72
HTLM-Auto-NS
27.56/10.17/24.57
33,40/13
6.71/1.98/7.88
15.15/2.54/10.91
HTLM-Auto-S
28.73/11.11/6.499 34.65/14.54/32.15
8.15/2.92/9.75
17.14/3.41/13.4
HTLM-Manual
31.61/10.80228660
38.51/16.10/33.88
15.81/2.98/10.54
22.34/4.12/14.66
Table 1: HTLM results on zero-shot summarization. HTT.M-Manual denotes manually engineered
prompts with size hints, while HTLMMAuto-S and HTT.M-Att-NN indicate autoprompting with and without size hints respectively. Metrics shown are ROUGE-/RROTGE-/ROUUGL respectively.
FAR
WANLI
DART
BLEU
NIST
MEE
R.L. CIDER
BLEU
MET U
TER
BLEU
MEE
TER
Mover
BERT
BLEURT
S
S
Pine tuning
68.2
5.62
16.2
71.0
2.47
64.2
27.7
46.5
0.05
0.30
0.38
0.35
1.76
0.55
46.2
0.59
0.46
0.50
0.94
0.39
GPT-2
69.5
5.75
46.0
69.9
2,45
65.3
43.1
55.5
0.66
0.35
0.42
0.35
1.55
0.42
47.0
0.39
0-46
0.51
0.94
9.40
HTIN
70.3
8.90
46.3
70.8
2.47
68.4
48.4
55.6
0.46
0.39
0.42
0.35
1151
0.40
47.2
0.39
0.44
0.51
0.94
0.40
Prefi (0.1%)
GPP DernuM
09-7
46.1
21.4
2.49
62.9
45.6
0.64
0.35
0.35
0.41
0.38
0.46
0.50
0.94
0.19
GPT2 soow
70.3
66.
71.7
2.47
63.4
47.7
0.39
0.34
0.48
46
6.99
0.40
70.1
2.45
Bose lites
PTL---Peeline
64.34
20.65
35,29
0.21
0.30
0.48
0.64
0.55
UII-VNU-Pipeein
19.57
0.11
7.07
0.15
0.03
0.09
0.78
087
0.82
Table 2: We evaluate GPT-2MEDIUM, GPT-2LARGE and HTLM on table-to-text generation on E2E (left), WebNLG (middle) and DART (right).
HTLM with manual prompts LLTLM-Manual) and size hints substantially improves state-of-lee the- art zero-shot summarization results on all four datasets without any tailored pretraining. In particular, see large improvements of more than 8 ROUGE-L F1 for the Gigaword dataset. Furthermore, size hints-based auto-prompting (ETLM-Auto-S outperforms PEGASUS in three out of four datasets. Specifically, for the Gigaword dataset, we outperform previous faateoff-the-ar zero-shot results from PEGASUS by roughly 6 ROUGE-L points, HTLM improvements stem from the fact that HTML-bbased prompting allows us better control over sapaset- attributes such as length and style.
For NLG tasks, we required the use of a single training example to get prompting to work sufficiently. We report these one-shot numbers Table 2 Because these tasks require structured tabular inputs, is not obvious how to prompt any other text-based pre-trained models. We report other non-trainable baselines such as the grammar based pipeline approaches (TILB/UIT-VE VNU) Gardent et al. (2017).
To the best of our knowledge, these are the first one-shot table to text, natural language generation
results.
4.2 CLASSIFICATION
For prompting in the classification setting, we select 4 datasets to work with. Instead of relying on generative prompting to generate target token(s) denoting the correct class, we instead rely on perplexity measures over the set of all targets to select the correct class. In other words, we select the class for which the perplexity of the corresponding instantiated template the smallest.
RTE (Bentivogli et al., 2009) is a textual entailment task formulated as binary classification. We place the candidate <div> element with the class attribute set candidate and do the same with the respective hypothesis. In the third element, we utilize the prompt from Brown et al. (2020) with the class attribute set to answer.
BoolQ (Clark et al. 2019) is a yes/no question answering task, also formulated as binary classification for question, passage, and answer triplets. We represent the question <di v> element with the

---

Under review as a conference paper at ICLR 2021
itemprop set to https://schema... wwtpaemmmm.r///ueettoon passage as div element with class attribute passage and answer as a div element with the itemprop set to https//schemaa.org/asseerr
Winogrande (Levesque et al. 2012) consists of adversarially collected Winograd Schema Chal- lenge Levesque al. 2011) data. We utilize the same template GPT-3 but place it in QA style template similar to BoolQ. Please refer to the Appendix for exact templates.
HellaSwag The last dataset we evaluate is the commonsense natural language inference task Hel- laSwag which, due to its adversarial nature, is considered complex (Zellers et al. 2019).
We present results zero-shot classification in Table 3
HTLM prompting of classification datasets outperforms the most comparable (in terms of number of parameters) GPT-3 Medium sized model on the majority of tasks, while approaching and on RTE outperforming- the GPT-3 Large model which consists of roughly double the amount parameters as HTLM.
RTE
BoolQ
Winogrande
HellaSwag
Params
GPT-3
63.5
60.5
70.5
78.9
175B
GPT-3 Large
48.4
58.9
57.4
51.0
760M
GPT-3 Med
49.8
60.3
52.1
43.6
350M
HTLM-Manual
51.2
55.3
54.8
47.9
400M
Table 3:
Classification accuracy with zero shot prompting. We compare our performance to the full GPT-3 model as well as variants of comparable size.
FINE-TUNING EXPERIMENTS
In addition to our previous prompting results, we also aim to show that HTLMM learned representations useful in the full finetuning setting least as well other pre-trained representations. We compare against other pre-training MLM models such as RoBERTa Liu et al. (2019), original BART Lewis al. 2019), and T5 Raffel
(0099) by finetuning on the GLUE benchmark (Wang et al. 2018).
During finetuning, instead of a simple concatenation of sentences from the train set, we place the examples into prompts derived from Le Scao & Rush (202T). We defer the Appendix for the exact prompts. Given the recent advancements in finetuning, we also report results using the recently proposed R3F method for finetuning (Aghajanyan al. (220a)) for both RoBERTa and HTLM. We note that there are other numerous pre-training methods we do not compare against (such as Devlin al. (2018); Clark al. (2020); : He al. (2020); Meng et al. (202T) but the goal of our fine-tuning results is to show that there is no significant degradation in quality from training structured documents.
MNLI Acc-m/mm
QQP
RTE
QNLI
MRPC
CoLA
SST-2 Acc
# Params
Acc
Acc
Acc
Acc
Mcc
RoBERTA RoBERTa-R3F
90.2/-
92.2
86.6
94.7
89.1
68.0
96.4
330M
91.1/91.3
92.4
88.5
95.3
91.6
71.2
97.0
330M
T5-Base
87.1/86.2
89.4
80.1
93.7
87.5
51.1
95.2
220M
T5-Large
89.9/89.6
89.9
87.2
94.8
89.9
61.2
96.3
770M
BART-Large
89.9/90.1
92.5
87.0
94.9
90.4
62.8
96.6
400M
HTLM
90.3/91.4
92.6
87.1
95.1
90.8
64.3
96.9
400M
HTLM-R3F
91.4/92.1
92.8
89.1
95.4
91.5
69.4
97.1
400M
RTIMM-RFFPrropt
91.6/91.2
92.9
89.4
95.7
91.7
69.8
97.3
400M
Table 4: Results on the GLUE development set for various fine-tuning methods applied to HTLM.
We present results in Table Overall HTLM is competitive with other pre-training methods. We also note that we can improve fine-tuning performance by placing the examples into prompts and fine-tuning the classification head. The improvements that see in terms of prompting have
7

---

Under review as a conference paper at ICLR 2021
adverse effects fine-tuning but are rather positive, providing further evidence that the proposed approach of structured pre-trainn pre-training is a viable alternative to other methods of pre-traine pre-training even for fine-tuning.
We also show our fine-tuning results for the table-to-text generation datasets in Table Similar to GLUE fin-tuning, we place all NLG samples into prompt while fine-tuning. HTLM fine-tuned is able to outperform both variants of the GPT-2 model consistently.
6 PROMPT DATA EFFICIENCY
Average Advantage (# Training Points, P vs. H)
MNLI
BoolQ
CB
RTE
WiC
RoBERTa-Large
3506 + 536
752 + 46
90 + 2
282 + 34
-424 = 74
Ta-Large
5010 + 230
650 +
150 +
300 + 65
-220
BART-Large
4020 + 220
450 = 55
125 + 10
305 + 25
-110 = 45
HTLM
6025 + 440 855 + 205
255 + 35
840 + 45
45 + 25
Table 5: Average advantage (higher is better) in terms of training points for fine-tuning well-structured prompt (P) against classical classification head (H).
Average Advantage (# Training Points, P vs. N) BoolQ
MNLI
CB
RTE
WiC
RoBERTa-Large
150 + 252
299 81
78 + 2
404 + 68
-354 + 166
T5-Large
300 + 120
350 + 95
150 + 4
608 + 90
20 + 43
BART-Large
200 + 180
325 + 54
85 + 8
512 + 64
-80 + 89
HTLM
692 - 240
565 + 143 255 + 34
640 + 45
80 + 40
Table 6: Average advantage (higher better) terms of training points for fine-tuning gell-structured prompt (P) against prompt with non-sensical verbalizer (N).
What does the HTMLL- pretraining and prompting scheme offer over one based on the plain text? Le Scao & Rush (202I) explored quantifying how many data points single prompt worth. Specifically, they analyzed three different -seeiicc settings given a pattern (the structure that the inputs are put into) and verbalizer (i.e., yes/no answer to pattern): (1) fine-tuning classification head (H), (2) fine-tuning the verbalizer a prompt encoding the semantics of the task (P), and (3) fine-tuning the prompt but with verbalizer that is non-sensical (N).
By carefully selecting the number of data points to be used during training in each setting while matching the end fine-tuning performance, terms of data points. We provide the same analysis extended to BART, T5-Large, and HTLM using the same PET prompts provided in SShiick & Schütze 2020)). For HTLM, we wrap all PET prompts in an HTML element. We select the same datasets that were used in the original paper for our experimentation; MNLI (Williams al. 2018), BoolQ (Clark et al. 2019), CB De Marneffe et al. (2019). RTE (Bentivogli et al.) 2009). WiClPilehvar & Samacho-Colladee
can empirically measure the efficacy of prompts in
(2019).
We first look at the average advantage of fine-tuning a prompt (P) against a classification head (H) in Table 5 We see that across the board, HTLM prompts-i.e hypertext prompts applied to HTIMMare worth more than natural language prompts to various other pre-trained models. Compared smaller datasets, HTLM's advantage is close to triple on CB and double on RTE. Furthermore, on WiC, HTLM is the only pre-trained model capable of having a positive training advantage when using prompts. We view this additional evidence the benefit of pre-training structured data on the prompting of pre-trained model.
to RargERTa-aage
We also compare the average advantage of fine-tuning prompt with verbalizer (P) that makes sense against against finetuning a prompt where we change the verbalizer to a random first name (N). This is important capture whether the benefits arise from representing the data in their respective
8

---

Under review as a conference paper at ICLR 2021
patterns the coupling of the pattern and the verbalizer. We present our results in Table Relative to the previous P vs. H setting we lose a large amount of advantage, as was similarly seen in (Le Scao & Rush 202I]. Interestingly enough for small datasets such CB, all of the training advantage of the prompt comes from the pattern in HTLM
We view this as further evidence that structured, document level approach to both pre- pre-training and prompting can be seen as a viable alternative to a purely natural language approach.
7 RELATED WORK
GPT-2 Radford et al. (019) showed that large language models show varying levels of zero-shot
performance across NLP tasks when compared to supervised baselines (e.g., rudimentary performance on summarization, but more competitive results on reading comprehension). Brown et al. (2020) through their GPT3 model showed that by further scaling up language models a large subset of the internet, prompting could be a viable alternative to standard finetuning. The success of GPT3 was largely attributed to massive size and compute-intensing pretraining. By reformulating NI.P tasks as cloze-style questions, [Schick& & Schütze (2020) shows that the prompting capabilities exhibited by GPT3 can occur in language models of a much smaller scale when gradient-based finetuning is combined with task-specific unlabeled data. Follow-up work Tam et al. (2021) improves upon these results without depending on unlabeled data. Unlike GPT-3 and other models which use conventional natural language text-based prompting, we focus on a new hyper-text based prompting scheme using generative masked language models pre-trained directly over HTML.
For task-specific zero-shot performance, custom pre-training and data augmentation schemes have been developed. For example, PEGASUS (Zhang et al. (209)) proposes novel pre-training scheme tailored for summarization which involves masking and generating salient gap sentences from large news corpus. While PEGASUS is capable of doing zero-shot summarization, offers little control summary attributes such as length and style which vary across different summarization datasets. WikiTraanser Fabbri et al. (02T)) fine-tunes pretrained models on pseudo-summan summaries, produced from generic Wikipedia data, which contain characteristics of the target dataset, such the length and level of abstraction. Our proposed model allows fine-grained control over the length of the generated text by specifying the size of the mask. Moreover, by using different prompts, HTLM can produce stylistically varied summaries without dataset-specific augmentation and finetuning.
Another line of work has been looking at hybrid form of prompting that attempts to optimize very few parameters to solve an end task. For example Li & Liang 20221) argue that optimizing in the continuous prompt space is an effective solution to prompt search while Aghajanyan al. (2020b) optimize for low-rank projection of the full parameter space. For simplicity, we only focus on either full-finetuning or zero-shot prompting this paper.
Attempts have been made to encode architectural priors for structured inputs into transformers as well. Specifically, Ainslie et al. (2020) discuss new type of model which allows for scalability in input length as well as the ability to encode the structure of the input. We opt to allow HTLM to learn the structure that available in the HTML directly without encoding any structural priors into the model itself.
8 CONCLUSION
In this paper, we proposed HTLM, hyper-text language model trained on simplified HTML documents from large-scale web crawl. We showed that by directly modeling HTML through BART-like objective, we could do structured zero-shot prompting by representing tasks in HTMI.. Specifically, we outperform the previous best results on zero-shot prompting for summarization by a wide margin by creating prompts that capture the underlying semantics of cach summarization dataset. Furthermore, we show that pre-training on structured data improved full finetuning performance relative to other pre-trained models that only modeled natural language.
We also showed additional advantages of modeling hyper-text, beyond improved accuracy. HTLM can be used for prompt by simply asking the model to recover the document structure from training samples; these auto- pootops on datasets like Gigaword and CNN/DM outperformed previous state-of-fe of-e-oof-the-at zero-shot approaches. Lastly, provided an in-depth comparison of the training

---

Under review as a conference paper at ICLR 2021
advantage, in terms of data efficiency, that HTLM had compared to other pre pre-training approaches. Across the board, HTML prompts were worth more to HTLM than natural language prompts were worth to our baselines, further showing the efficacy of pre-training structured data.
Future work can focus on the scaling laws of structured pre-training and prompting. As was seen from GPT-3, the size of the model and the amount of compute utilized and significant impact prompting performance.
REFERENCES
Armen Aghajanyan, Akshat Shrivastava, Anchit Gupta, Naman Goyal, Luke Zettlemoyer, and Sonal Gupta. Better fine-tuning by reducing representational collapse. arXiv preprint arXiv:2008.03/56, 2020a.
Armen Aghajanyan, Luke Zettlemoyer, and Sonal Gupta. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. arXiv preprint arXiv:2012.133255, 2020b.
Joshua Ainslie, Santiago Ontanon, Chris Alberti, Vaclav Cvicek, Zachary Fisher, Philip Pham, Anirudh Ravula, Sumit Sanghai, Qifan Wang, and Li Yang. Etc: Encoding long and structured inputs in transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 268 284, 2020.
Iz Beltagy, Matthew Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.057 1.00000.0555..... 2020.
Anja Belz and Ehud Reiter. Comparing automatic and human evaluation of nlg systems. In //th conference of the european chapter of the association for computational linguistics, 2006.
Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. The fifth pascal recognizing textual entailment challenge. In TAC, 2009.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165. 2020.
Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. arXiv preprint arXiv:2009. 14794, 2020.
Christopher Clark, Kenton Lee, Ming-Wes Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. Proceedings of NAACL-HLT 2019, 2019.
Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003. 10555, 2020.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Fran- cisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross-lingual representation learning at scale. arXiv preprint arXiv: /91 1992111116,,,.111111100111011111 2019.
Marie-Catherine De Marneffe, Mandy Simons, and Judith Tonhauser. The ConmitmentBank: Investigating projection in naturally occurring discourse. 2019. To appear in proceedings of Sinn und Bedeutung 23. Data can be found at
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1rX.v:1810.048005 2018.
A. R. Fabbri, Simeng Han, Haoyuan Li, Haoran Li, Marjan Ghazvininejad, Shafiq R. Joty, Dragomir Radev, and Yashar Mehdad. Improving zero and few-shot abstractive summarization with interme- diate fine-tuning and data augmentation. In NAACL, 2021. 2021.
Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Perez-Beltranee Beltraceli.. The webnlg challenge: Generating text from rdf data. In Proceedings of the 10th International Conference on Natural Language Generation, pp. 123-133, 2017.
10

---

Under review as a conference paper at ICLR 2021
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhanced enhanced bert with disentangled attention. arXiv preprint arXiv:2006.03654, 2020.
Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In Advances in neural information processing systems, pp. 1693-1701, 2015.
Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, Hérve Jégou, and Tomas Mikolov. Fasttext. zip: Compressing text classification models. arXiv preprini arXiv: 1612.0361, 2016.
Byeongchang Kim, Hyunwoo Kim, and Gunhee Kim. Abstractive summarization of reddit posts with multi-level memory networks. arXiv preprint arXiv: 2018.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. 2014.
Alon Lavic and Abhaya Agarwal. Metcor: An automatic metric for mt evaluation with high levels of correlation with human judgments. In Proceedings of the second workshop on statistical machine translation, pp. 228-231, 2007.
Teven Le Scao and Alexander Rush. How many data points is a prompt worth? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2627-2636, Online, June 2021. Association for Computational Linguistics. doi: 10. 10.18653/v1/20211 maacI-main.20S. URL https: /www
ac aclweb.
naacc main. 208
Hector Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. In Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning. Citeseer, 2012.
Hector Levesque, Ernest Davis, and Lcora Morgenstern. The Winograd schema challenge. AAAI Spring Symposium: Logical Formalizations of Commonsense Reasoning, volume 46, pp. 47, 2011.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising natural language generation, translation, and comprehension. arXiv preprint 19..v: 2019.
pre-training for
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.000 00010000000,,, 2021.
Chin-Ye Yew Lin. Rouge: A package for automatic evaluation of summarics. In Text summarization branches out, pp. 74-11, 2004.
Yinhan Liu. Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanow. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: /907. 1692, 2019.
Yu Meng, Chenyan Xiong, Payal Bajaj. Saurabh Tiwary, Paul Bennett, Jiawei Han, and Xia Song. Coco-Im: Correcting and contrasting text sequences for language model pretraining. arXiv preprint 1oXiv:2/02.08473 2021.
Linyong Nan, Dragomir Radev, Rui Zhang, Amrit Rau, Abhinand Sivaprasad, Chiachun Hsieh, Xiangru Tang, Aadit Vyas, Neha Verma, Pranav Krishna, et al. Dart: Open-domain structured data record to text generation. arXiv preprint arXiv:2007.02881,, 2020.
Courtney Napoles, Matthew R Gormley, and Benjamin Van Durme. Annotated gigaword. In Proceedings of the Joint Workshop on Automatic Knowledge Base Construction and Web-scale Knowledge Extraction (AKBC-WEKEN) pp. 95-100, 2012.
Shashi Narayan, Shay B Cohen, and Mirella Lapata. Don't give me the details, just the sum- mary! topic-aware convolutional neural networks for extreme summarization. arXiv preprint arXiv: 1808 08745, 2018.
11

---

Under review as a conference paper at ICLR 2021
Jekaterina Novikova, Ondiej Dušek, and Verena Rieser. The e2e dataset: New challenges for end-to-end generation. arXiv preprint arXiv:1706.09230 2017.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
Ethan Perez, Douwe Kiela, and Kyunghyun Cho. True few-shot learning with language models. arXiv preprint arXiv:2105.1 1447, 2021.
Mohammad Taher Pilchvar and Jose Gamacho-Colladee WIC: The word-in indtint- dataset for evaluating context-sensitive meaning representations. In Proceedings of NAACL-HLT, 2019.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl Blog, 1(8)99 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv: 10683, 2019.
Timo Schick and Hinrich Schütze. It's not just size that matters: Small language models are also few-shot learners. arXiv preprint arXiv:2009.07116 2020.
Mathew Snover, Bonnie Dorr, Richard Schwartz, John Makhoul, Linnea Micciulla, and Ralph Weischedel. study of translation error rate with targeted human annotation. In Proceedings of the 7th Conference of the Association for Machine Translation in the Americas (AMTA 06), pp. 223-231, 2005.
Derek Tam, R. R. Menon, M. Bansal, Shashank Srivastava, and Colin Raffel. Improving and simplifying pattern exploiting training, ArXiv,
2021.
Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus- image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4566-4575, 2015.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 353-355, Brussels, Belgium, November 2018. Association for Computational Linguistics. doi:
10.11663/11/111554444 URL htLLhhtps:///wtt aclweb. org aataaaotohohhaaaa 5446
Sinong Wang, Belinda Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint 2.Xi::0006.04768. 2020.
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo- gies, Volume / (Long Papers). pp. 1112-1122. Association for Computational Linguistics, 2015 8. URL httthttp://aaat www.wwwwwwbb...m.. org/ wttpee/.......aaaoooooo//---1111
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint 195.:2190 2019.
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. arXiv preprint arXiv:1912.08777 2019.
A APPENDIX
A.1 FINETUNING HYPER-PARAMETER
For our GLUE related experiments the following parameters are used.
12

---

Under review as a conference paper at ICLR 2021
Hyper Parameter
MNLI
QNLI
QQP
SST-2
RTE
MRPC
CoLA
Learning Rate
5e-6
5e-6
5e-6
5e-6
le-5
le-5
le-5
Max Updates
123873
33112
113272
20935
3120
2296
5336
Max Sentences
8
32
32
8
16
16
Table 7: Task specific hyper parameters for GLUE experiments
Hyper parameter
Value
Optimizer
Adam
Hyper parameter
Value
Adam-beta
(0.9, 0.98)
Adam-eps
le-6
[0.1, 0.5, 1.0, 5.0] [U, le - 5
LR Scheduler
polynomial decay
Noise Types
Dropout
0.1
Weight Decay
0.01
Warmup Updates
0.06 * max updates
Table 8: Hyper parameters for R3F experiments GLUE
13


Reviews:
[{"review_id": "iIdpKW5YTmI", "reviewer": "Reviewer_36N9", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strength\n1. the writing is mostly linear and easy to follow if the reader is familiar with the field, especially, BART (see additional comments below on writing).\n2. the authors are transparent about how to replication process (data preprocessing, training tricks, hyper parameters, etc.) and the authors pledge to release their work.\n3. the proposed model seems to be a strong zero/one-shot learner (both text generation, and classification) and appears to be on par with strong existing baseline.\n4. the HTLM model can also be used via the more traditional pretrain+finetune paradigm and achieve good performance on GLUE set.\n5. the authors proposed a way to create the so-called auto-prompt, which can rival hand-crafted prompts (see additional questions below) \n\n## Weakness and questions\n\n1. the writing is inconsistent in several places and makes the related sections difficult to understand.\nSec 3.1 describes manual-NS and manual-S. But I can't find these settings in the experiments. in the experiments, HTLM-Manual is the only manual setting.\nAlso, related to writing, Table 2 font is really small and hard to read and is inconsistent with the rest of the paper.\n\n\n2. The pretraining data, after cleaning is 23TB, which is a lot for retraining. So I looked up the amount of text data used in RoBERTa and BERT pretraining. According to https://arxiv.org/pdf/1907.11692.pdf , RoBERTa uses up to 0.16TB and BERT uses 0.013TB. So the proposed model is using more than 140x data than RoBERTa and more than 1,700x data than BERT. I wonder if there is any correlation between the amount of pretraining data vs the zero shot or finetune accuracy? what would happen to HTLM if it only had 0.013TB pretraining data instead of 23TB?\n\n3. Fig 2 attempts to illustrate the idea of auto-prompting, which I'm a little confused about. In order the generate the template on the right, one has to get the pair of two paragraphs  and insert the four mask tokens around them. So in order to generate one prompt for the summarization task, the user has to prepare this one pair of sample. Is Auto-prompting zero-shot learning or in this case one-shot method? the authors titled section 4 as \"zero/one-shot prompting\" and the HTLM entry in Table 2 is under one-shot. but there is no further explanation if this is what the authors meant by one-shot.\n\n4. Also in figure 2, after the HTML is generated on the right. Is there additional processing needed for using this prompt. Based on Figure 2. the model generates not only the HTML tags but also addition text: ` | The Washington Post`. so despite it being called auto-prompt, the user still has to verify if the model is generate 1)  syntacticly correct HTML tags and 2) if the model is generating additional/unwanted text. As I understand it, if the model generates this additional ` | The Washington Post` and the human user doesn't catch it, this text will be part of the prompt and be added to all samples during the testing , which presumably would be bad for performance. So now that it requires human verification of HTML syntactic correctness and any potential text hallucination, can we still call it \"auto\"?\n\n5. What denoising pre training tasks are used exactly?\nIn intro and Figure 1, the authors mention title-tag denoising and class/id attribute denoising as examples for the model to pre-train with. Are there any other tasks the model is trained on? for example, there can be list and tabular elements in HTML. Does the model train with those structures?\n\n\n\n\n\n", "summary_of_the_paper": "This paper presents HTML-based large scale language model pretraining. The model architecture and pre-training method are both based on BART. The author presents the resulting model, HTLM, as a strong zero-shot learner while being on par with other large transformer language models on the GLUE fine tune tasks. the authors show that via various benchmarks in their experiment section.", "summary_of_the_review": "This paper presents HTML-based large scale language model pretraining. The model architecture and pre-training method are both based on BART. The author presents the resulting model, HTLM, as a strong zero-shot learner while being on par with other large transformer language models on the GLUE fine tune tasks. \n\nWhile the authors present a lot of details in a linear fashion, there are still several gaps in the writing and would require clarification.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "UU-YcB3Kh3E", "reviewer": "Reviewer_srNC", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I would like to thank the authors for this intriguing piece of work – I particularly enjoyed reading the rigorous analyses of the proposed language model and am curious to see how the artifacts of this research will be leveraged by the broader NLP community. I would summarize the strengths and weaknesses of this paper as follows:\n\nStrengths:\n1. The idea of leveraging HTML text for pre-training large language models, and linking it to the structure of documents for supervision is quite interesting and creating. As the authors demonstrate via zero-shot summarization experiments, such a pre-training can be beneficial for specific NLP tasks like summarization. \n2. The experiments in the paper are conducted on the GLUE benchmark. The analysis and experiments are rigorous and support the claims made in the paper. The paper is well-written and the contributions are easy to comprehend.\n3. I particularly like the analysis that aims to understand the per-prompt-efficacy of HTLM and contrasts it with that of existing pre-trained language models. \n\nWeaknesses:\n1. The authors do not conduct their experiments on SUPERGLUE – a much more difficult benchmark than GLUE. This is in contrast to pre-trained language models like T5 that were evaluated on both GLUE and SUPERGLUE tasks. This seems to be an important evaluation that is currently missing from the proper. \n2. It was hard for me to comprehend some of the choices made in the paper. For instance, the decision to skip MHTML documents whose ratio of text to HTML was not greater than 0.46 did not seem well justified to me. Perhaps explaining the filtering strategy in Section 2.1 would help. In some of the cases, it was not clear what the size hint was and how it was determined (e.g., in Table 1 for auto-prompting with size hints included).\n3. Since the concept of size hint is one of the key contributions of this work, it would have helped to see how sensitive the results are with respect to various size hints (sensitivity of the downstream to +/-1 or 2 sizes). This analysis is currently missing from the paper.\n\nOverall, I enjoyed reading the work. Since the contribution of this work hinges on rigorous evaluation, I would primarily encourage the authors to expand their evaluations to include the SUPERGLUE tasks, as some of the previous studies do. \n", "summary_of_the_paper": "The paper proposes a new large language model, called HTLM, short for Hyper-Text Language Model. The language model has been trained on 25TB of HTML text. The authors argue that HTML tags provide valuable information regarding document-level structure. Besides the scale, the authors also adopt an optional pre-training strategy where it is possible to provide size hints to govern the length of the generated text or each mask (alongside the BART-like training objective). The trained language model achieves state-of-the-art performance on zero-shot summarization by prompting the model to predict the text within the <title> tags. The method also achieves state-of-the-art performance on several other benchmark NLP tasks, performing better than large pre-trained models that were only trained on text data (as opposed to hyper-text data). The authors also discuss manual and automated prompt construction, and their evaluation shows that the model can also perform better than competitive baselines on the multiple classification tasks. A follow-up analysis also shows that the per-prompt-efficacy is higher for HTLM than for text-only language models. ", "summary_of_the_review": "The work is technically sound, the core idea is creative, and involves a series of rigorous evaluations to substantiate the claims made. However, in line with some of the highly related previous works, I believe that the paper will benefit from evaluations on some additional NLP tasks that are considered more difficult (SUPERGLUE). Also, adding a few missing details and conducting a sensitivity analysis of the downstream performance with respect to the size hint would help the readers and practitioners who would use the pre-trained model for downstream tasks. I recommend a weak acceptance for the paper. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Rfb3I5nM11q", "reviewer": "Reviewer_Pgkk", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I really enjoyed reading this paper. The contribution is primarily empirical and is thorough. Modeling the HTML directly seems sensible and highly practical given the large amount of web data that can be used and all the organization that is already done in web-pages for applications like SEO. Things like the <title> tag seem to provide extra annotation to the text and the model seems to utilize these in summarization tasks. Overall I have nothing to complain about and think this is going to be a rich and interesting line of work - training LMs directly on structured text data.\n\nI am curious if the authors performed any multi-lingual analysis on benchmarks like xtreme (https://arxiv.org/abs/2003.11080) - could be a potential future direction.", "summary_of_the_paper": "This paper introduces a large-scale LM trained directly on the raw HTML in a large-scale web crawl (the common crawl corpus). The resulting model is able to utilize the structure in HTML documents for a variety of tasks such as zero shot summarization, fine-tuning, classification and more. It looks like using structured data for pre-training and creating prompts provides a variety of advantages in LMs. The paper has a comprehensive set of experiments and a thorough ablation study.", "summary_of_the_review": "The paper presents a straightforward idea to pre-train large scale LMs directly on HTML. The empirical analysis is thorough, the results look very good, and the writing flows very nicely. Overall a very nice and fun read.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "9ltk6a1YIw8", "reviewer": "Reviewer_2QhH", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1. This paper studies a new direction in language model pretraining that goes beyond using plain text as the pretraining data. It effectively leverages the HTML data which can be obtained in large amounts via common crawl.\n2. The paper is overall well-written with sufficient details and organized presentations.\n3. The proposed model achieves superior performance on zero-shot summarization and does well on classification tasks.\n\nCons:\n1. The technical contribution and novelty are rather weak. The model architecture and training objectives are largely based on BART, with some modifications to tailor for HTML-format training data.\n2. The HTLM model is pretrained from a BART-large checkpoint which means it still needs to start from a language model that is trained on plain texts. It's unclear how necessary is this dependency (e.g. if HTLM is randomly initialized instead of continuing training from BART, will the results be much worse?)\n3. Some presentations can be further improved. For example, it would be better to include an illustrative figure or explicit formulas showing an overview of how the BART pretraining objectives are tailored to the HTML-format training.\n4. The full fine-tuning experiments in Sections 5 and 6 fail to cover recent state-of-the-art plain text pretrained models, like ELECTRA [2], DeBERTa [3] and COCO-LM [4]. And the statements that \"HTLM improves over existing pre-training methods\" and \"hyper-text prompts provide more data efficiency to the HTLM model than plain text prompts do for existing LMs\" seem to overclaim. Actually, the most recent plain text pretrained LM, COCO-LM, achieves better performance on GLUE than HTLM with fewer model parameters and has comparable performance with HTLM-R3F-Prompt which leverages additional fine-tuning techniques. I actually don't think HTLM (as the first PLM trained on large-scale HTML-format data) has to claim better performance than LMs trained on natural language texts on GLUE tasks to be impressive, but it is necessary to acknowledge that there exist better plain text pretrained LMs and give them credits.\n\nMissing References:  \n[1] @inproceedings{devlin2019bert,\n  title={BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding},\n  author={Devlin, Jacob and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina},\n  booktitle={NAACL-HLT},\n  year={2019}\n}  \n[2] @inproceedings{clark2020electra,\n  title={ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators},\n  author={Clark, Kevin and Luong, Minh-Thang and Le, Quoc V and Manning, Christopher D},\n  booktitle={ICLR},\n  year={2020}\n}  \n[3] @inproceedings{he2021deberta,\n  title={DeBERTa: Decoding-enhanced bert with disentangled attention},\n  author={He, Pengcheng and Liu, Xiaodong and Gao, Jianfeng and Chen, Weizhu},\n  booktitle={ICLR},\n  year={2021}\n}  \n[4] @inproceedings{meng2021coco,\n  title={COCO-LM: Correcting and contrasting text sequences for language model pretraining},\n  author={Meng, Yu and Xiong, Chenyan and Bajaj, Payal and Tiwary, Saurabh and Bennett, Paul and Han, Jiawei and Song, Xia},\n  booktitle={NeurIPS},\n  year={2021}\n}\n\nQuestions:\n1. It is mentioned in the abstract and introduction that \"element class and id attributes can encode categorical properties of documents\". I'm not completely clear how these types of information are being leveraged in HTLM pretraining. Is the model explicitly trained to predict document class or ids?\n2. HTLM has great performance on zero-shot summarization. Have you tried few-shot or even full training summarization? \n3. The paper seems to mention some concrete examples/prompts are included in the Appendix, but I cannot find them.\n\n\n", "summary_of_the_paper": "This paper introduces HTLM which is a language model pretrained on a large-scale web crawl hyper-text data. There are several contributions in the paper:\n* A preprocessing step to filter out noisy components in the web pages is proposed. The resulting simplified format, Minimal-HTML (MHTML), is likely to be composed of high-quality documents which can be used for pretraining.\n* A modified BART pretraining objective is proposed to inject noisy size hints to control the length of the span to be generated by the model during training.\n* A new prompting method in the form of HTML templates is described to accomplish generation (e.g., summarization & table-to-text) and classification (e.g., GLUE) tasks.\n* The resulting pretrained HTLM model has superior performance on zero-shot summarization and can do better than some existing language models pretrained on plain texts.", "summary_of_the_review": "Overall, I appreciate the authors' efforts to propose the first PLM trained with HTML-format data and make it work. The pretrained model also shows impressive performance on some tasks (e.g., zero-shot summarization). However, I have some concerns about the paper (e.g., weak technical contribution and novelty, dependence on plain text PLMs, overclaims in the comparisons with state-of-the-art plain text PLMs, missing important references). If the authors could address my concerns in the rebuttal, I'm willing to adjust my rating.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper develops a new large language model trained on 25TB of (simplified) HTML text data. The HTML tags provide valuable information about the document structure. The training adapted the BART denoising objectives (to inject noisy size hint to control generation length during training). The paper also studies various prompting methods for the model. The model achieves state-of-the-art performance on zero-shot summarization and several text classification tasks. Reviewers have found the motivation of pretraining with structured text convincing, and the results are good.
Decision: Accept (Poster)