Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a new large language model, called HTLM, short for Hyper-Text Language Model. The language model has been trained on 25TB of HTML text. The authors argue that HTML tags provide valuable information regarding document-level structure. Besides the scale, the authors also adopt an optional pre-training strategy where it is possible to provide size hints to govern the length of the generated text or each mask (alongside the BART-like training objective). The trained language model achieves state-of-the-art performance on zero-shot summarization by prompting the model to predict the text within the <title> tags. The method also achieves state-of-the-art performance on several other benchmark NLP tasks, performing better than large pre-trained models that were only trained on text data (as opposed to hyper-text data). The authors also discuss manual and automated prompt construction, and their evaluation shows that the model can also perform better than competitive baselines on the multiple classification tasks. A follow-up analysis also shows that the per-prompt-efficacy is higher for HTLM than for text-only language models. 
main_review: I would like to thank the authors for this intriguing piece of work – I particularly enjoyed reading the rigorous analyses of the proposed language model and am curious to see how the artifacts of this research will be leveraged by the broader NLP community. I would summarize the strengths and weaknesses of this paper as follows:

Strengths:
1. The idea of leveraging HTML text for pre-training large language models, and linking it to the structure of documents for supervision is quite interesting and creating. As the authors demonstrate via zero-shot summarization experiments, such a pre-training can be beneficial for specific NLP tasks like summarization. 
2. The experiments in the paper are conducted on the GLUE benchmark. The analysis and experiments are rigorous and support the claims made in the paper. The paper is well-written and the contributions are easy to comprehend.
3. I particularly like the analysis that aims to understand the per-prompt-efficacy of HTLM and contrasts it with that of existing pre-trained language models. 

Weaknesses:
1. The authors do not conduct their experiments on SUPERGLUE – a much more difficult benchmark than GLUE. This is in contrast to pre-trained language models like T5 that were evaluated on both GLUE and SUPERGLUE tasks. This seems to be an important evaluation that is currently missing from the proper. 
2. It was hard for me to comprehend some of the choices made in the paper. For instance, the decision to skip MHTML documents whose ratio of text to HTML was not greater than 0.46 did not seem well justified to me. Perhaps explaining the filtering strategy in Section 2.1 would help. In some of the cases, it was not clear what the size hint was and how it was determined (e.g., in Table 1 for auto-prompting with size hints included).
3. Since the concept of size hint is one of the key contributions of this work, it would have helped to see how sensitive the results are with respect to various size hints (sensitivity of the downstream to +/-1 or 2 sizes). This analysis is currently missing from the paper.

Overall, I enjoyed reading the work. Since the contribution of this work hinges on rigorous evaluation, I would primarily encourage the authors to expand their evaluations to include the SUPERGLUE tasks, as some of the previous studies do. 

summary_of_the_review: The work is technically sound, the core idea is creative, and involves a series of rigorous evaluations to substantiate the claims made. However, in line with some of the highly related previous works, I believe that the paper will benefit from evaluations on some additional NLP tasks that are considered more difficult (SUPERGLUE). Also, adding a few missing details and conducting a sensitivity analysis of the downstream performance with respect to the size hint would help the readers and practitioners who would use the pre-trained model for downstream tasks. I recommend a weak acceptance for the paper.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper introduces a large-scale LM trained directly on the raw HTML in a large-scale web crawl (the common crawl corpus). The resulting model is able to utilize the structure in HTML documents for a variety of tasks such as zero shot summarization, fine-tuning, classification and more. It looks like using structured data for pre-training and creating prompts provides a variety of advantages in LMs. The paper has a comprehensive set of experiments and a thorough ablation study.
main_review: I really enjoyed reading this paper. The contribution is primarily empirical and is thorough. Modeling the HTML directly seems sensible and highly practical given the large amount of web data that can be used and all the organization that is already done in web-pages for applications like SEO. Things like the <title> tag seem to provide extra annotation to the text and the model seems to utilize these in summarization tasks. Overall I have nothing to complain about and think this is going to be a rich and interesting line of work - training LMs directly on structured text data.

I am curious if the authors performed any multi-lingual analysis on benchmarks like xtreme (https://arxiv.org/abs/2003.11080) - could be a potential future direction.
summary_of_the_review: The paper presents a straightforward idea to pre-train large scale LMs directly on HTML. The empirical analysis is thorough, the results look very good, and the writing flows very nicely. Overall a very nice and fun read.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces HTLM which is a language model pretrained on a large-scale web crawl hyper-text data. There are several contributions in the paper:
* A preprocessing step to filter out noisy components in the web pages is proposed. The resulting simplified format, Minimal-HTML (MHTML), is likely to be composed of high-quality documents which can be used for pretraining.
* A modified BART pretraining objective is proposed to inject noisy size hints to control the length of the span to be generated by the model during training.
* A new prompting method in the form of HTML templates is described to accomplish generation (e.g., summarization & table-to-text) and classification (e.g., GLUE) tasks.
* The resulting pretrained HTLM model has superior performance on zero-shot summarization and can do better than some existing language models pretrained on plain texts.
main_review: Pros:
1. This paper studies a new direction in language model pretraining that goes beyond using plain text as the pretraining data. It effectively leverages the HTML data which can be obtained in large amounts via common crawl.
2. The paper is overall well-written with sufficient details and organized presentations.
3. The proposed model achieves superior performance on zero-shot summarization and does well on classification tasks.

Cons:
1. The technical contribution and novelty are rather weak. The model architecture and training objectives are largely based on BART, with some modifications to tailor for HTML-format training data.
2. The HTLM model is pretrained from a BART-large checkpoint which means it still needs to start from a language model that is trained on plain texts. It's unclear how necessary is this dependency (e.g. if HTLM is randomly initialized instead of continuing training from BART, will the results be much worse?)
3. Some presentations can be further improved. For example, it would be better to include an illustrative figure or explicit formulas showing an overview of how the BART pretraining objectives are tailored to the HTML-format training.
4. The full fine-tuning experiments in Sections 5 and 6 fail to cover recent state-of-the-art plain text pretrained models, like ELECTRA [2], DeBERTa [3] and COCO-LM [4]. And the statements that "HTLM improves over existing pre-training methods" and "hyper-text prompts provide more data efficiency to the HTLM model than plain text prompts do for existing LMs" seem to overclaim. Actually, the most recent plain text pretrained LM, COCO-LM, achieves better performance on GLUE than HTLM with fewer model parameters and has comparable performance with HTLM-R3F-Prompt which leverages additional fine-tuning techniques. I actually don't think HTLM (as the first PLM trained on large-scale HTML-format data) has to claim better performance than LMs trained on natural language texts on GLUE tasks to be impressive, but it is necessary to acknowledge that there exist better plain text pretrained LMs and give them credits.

Missing References:  
[1] @inproceedings{devlin2019bert,
  title={BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding},
  author={Devlin, Jacob and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina},
  booktitle={NAACL-HLT},
  year={2019}
}  
[2] @inproceedings{clark2020electra,
  title={ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators},
  author={Clark, Kevin and Luong, Minh-Thang and Le, Quoc V and Manning, Christopher D},
  booktitle={ICLR},
  year={2020}
}  
[3] @inproceedings{he2021deberta,
  title={DeBERTa: Decoding-enhanced bert with disentangled attention},
  author={He, Pengcheng and Liu, Xiaodong and Gao, Jianfeng and Chen, Weizhu},
  booktitle={ICLR},
  year={2021}
}  
[4] @inproceedings{meng2021coco,
  title={COCO-LM: Correcting and contrasting text sequences for language model pretraining},
  author={Meng, Yu and Xiong, Chenyan and Bajaj, Payal and Tiwary, Saurabh and Bennett, Paul and Han, Jiawei and Song, Xia},
  booktitle={NeurIPS},
  year={2021}
}

Questions:
1. It is mentioned in the abstract and introduction that "element class and id attributes can encode categorical properties of documents". I'm not completely clear how these types of information are being leveraged in HTLM pretraining. Is the model explicitly trained to predict document class or ids?
2. HTLM has great performance on zero-shot summarization. Have you tried few-shot or even full training summarization? 
3. The paper seems to mention some concrete examples/prompts are included in the Appendix, but I cannot find them.



summary_of_the_review: Overall, I appreciate the authors' efforts to propose the first PLM trained with HTML-format data and make it work. The pretrained model also shows impressive performance on some tasks (e.g., zero-shot summarization). However, I have some concerns about the paper (e.g., weak technical contribution and novelty, dependence on plain text PLMs, overclaims in the comparisons with state-of-the-art plain text PLMs, missing important references). If the authors could address my concerns in the rebuttal, I'm willing to adjust my rating.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents HTML-based large scale language model pretraining. The model architecture and pre-training method are both based on BART. The author presents the resulting model, HTLM, as a strong zero-shot learner while being on par with other large transformer language models on the GLUE fine tune tasks. the authors show that via various benchmarks in their experiment section.
main_review: ## Strength
1. the writing is mostly linear and easy to follow if the reader is familiar with the field, especially, BART (see additional comments below on writing).
2. the authors are transparent about how to replication process (data preprocessing, training tricks, hyper parameters, etc.) and the authors pledge to release their work.
3. the proposed model seems to be a strong zero/one-shot learner (both text generation, and classification) and appears to be on par with strong existing baseline.
4. the HTLM model can also be used via the more traditional pretrain+finetune paradigm and achieve good performance on GLUE set.
5. the authors proposed a way to create the so-called auto-prompt, which can rival hand-crafted prompts (see additional questions below) 

## Weakness and questions

1. the writing is inconsistent in several places and makes the related sections difficult to understand.
Sec 3.1 describes manual-NS and manual-S. But I can't find these settings in the experiments. in the experiments, HTLM-Manual is the only manual setting.
Also, related to writing, Table 2 font is really small and hard to read and is inconsistent with the rest of the paper.


2. The pretraining data, after cleaning is 23TB, which is a lot for retraining. So I looked up the amount of text data used in RoBERTa and BERT pretraining. According to https://arxiv.org/pdf/1907.11692.pdf , RoBERTa uses up to 0.16TB and BERT uses 0.013TB. So the proposed model is using more than 140x data than RoBERTa and more than 1,700x data than BERT. I wonder if there is any correlation between the amount of pretraining data vs the zero shot or finetune accuracy? what would happen to HTLM if it only had 0.013TB pretraining data instead of 23TB?

3. Fig 2 attempts to illustrate the idea of auto-prompting, which I'm a little confused about. In order the generate the template on the right, one has to get the pair of two paragraphs  and insert the four mask tokens around them. So in order to generate one prompt for the summarization task, the user has to prepare this one pair of sample. Is Auto-prompting zero-shot learning or in this case one-shot method? the authors titled section 4 as "zero/one-shot prompting" and the HTLM entry in Table 2 is under one-shot. but there is no further explanation if this is what the authors meant by one-shot.

4. Also in figure 2, after the HTML is generated on the right. Is there additional processing needed for using this prompt. Based on Figure 2. the model generates not only the HTML tags but also addition text: ` | The Washington Post`. so despite it being called auto-prompt, the user still has to verify if the model is generate 1)  syntacticly correct HTML tags and 2) if the model is generating additional/unwanted text. As I understand it, if the model generates this additional ` | The Washington Post` and the human user doesn't catch it, this text will be part of the prompt and be added to all samples during the testing , which presumably would be bad for performance. So now that it requires human verification of HTML syntactic correctness and any potential text hallucination, can we still call it "auto"?

5. What denoising pre training tasks are used exactly?
In intro and Figure 1, the authors mention title-tag denoising and class/id attribute denoising as examples for the model to pre-train with. Are there any other tasks the model is trained on? for example, there can be list and tabular elements in HTML. Does the model train with those structures?






summary_of_the_review: This paper presents HTML-based large scale language model pretraining. The model architecture and pre-training method are both based on BART. The author presents the resulting model, HTLM, as a strong zero-shot learner while being on par with other large transformer language models on the GLUE fine tune tasks. 

While the authors present a lot of details in a linear fashion, there are still several gaps in the writing and would require clarification.
review_title: null

Meta-review: This paper develops a new large language model trained on 25TB of (simplified) HTML text data. The HTML tags provide valuable information about the document structure. The training adapted the BART denoising objectives (to inject noisy size hint to control generation length during training). The paper also studies various prompting methods for the model. The model achieves state-of-the-art performance on zero-shot summarization and several text classification tasks. Reviewers have found the motivation of pretraining with structured text convincing, and the results are good.
Decision: Accept (Poster)