Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work addresses the problem of using large language models for understanding HTML. Unlike prior work which attempt to solve this problem using dedicated architectures and training procedures and/or large HTML corpora, this work employs large language models pretrained on natural language text and evaluates their performance on three HTML understanding tasks - Semantic Classification of HTML elements, Description Generation for HTML inputs, and Autonomous Web Navigation of HTML pages, thus potentially eliminating the need for dedicated architectures and training procedures. Further, using only a small HTML corpus for finetuning a pretrained LM, the work reports encouraging results compared to LMs trained exclusively on the task dataset.


The key question asked by this work is can off-the-shelf LLM trained on a large text corpus be used in tasks that require some level of understanding of HTML. As canonical tasks in HTML understanding, the work looks at three tasks. 

In Semantic Classification, the ask from the model is to classify a salient HTML element into one of a set of role categories that are commonly used in automated form-filling applications. E.g. address, email, password. 

In Description Generation, the ask from the model is to, given a HTML snippet as the input, extract a small text sequence from the snippet as the natural language description of the snippet. 

In Autonomous Web Navigation, the ask from the model is to, given a HTML page and a natural language command as the input, identify the appropriate HTML elements and the actions on those elements that would satisfy the command. 


The work tests the idea of using pre-trained LLM for the three canonical tasks with several pretrained LLMs with different architecture  encoder-only, encoder-decoder, or decoder-only, different model size, and training data. Best results are obtained with encoder-decoder architectures with bi-directional attention.

The input to the models is the raw HTML text sequence. However, when the sequence is too big to fit into the context window of LLM, a snippet of appropriate size is extracted using a heuristic algorithm.

The work uses MiniWoB benchmark (demonstrations like email forwarding and social media interactions) for Autonomous Web Navigation task, a new dataset consisting of URLs from the real shopping websites for Semantic Classification, and a dataset derived from CommonCrawl for Description Generation.


strength_and_weaknesses: Strengths:
	1. That pre-trained natural language LLM can be effective for tasks involving HTML pages  is interesting and can potentially find use in several interesting practical applications.
	2. As no retraining of LLM with large HTML datasets is necessary, models for tasks  involving HTML pages can be developed quickly and less expensively. 
	3. That raw HTML text can be used as input without needing parsing is an advantage.
        4. Experimental results are very encouraging and validate the claim that pretrained LLMs can be effective for the three tasks.

Weaknesses:
        1. It is claimed that these three tasks require understanding of both structure and content of the web-page. While it is easy to see that textual content plays a key role in each of the three tasks, the role played by the structure of the web-page is not clear. It can be argued that no significant HTML structure analysis or understanding is needed for these tasks. For example, in Semantic Classification, what is most important for classifying HTML element 'input' into, say, 'username' is the value of its two attributes,  'type' and 'id'. As these attributes are in the close neighbourhood of 'input', parsing of HTML is not strictly necessary. Therefore, it might a good idea to do some experiments that demonstrate unequivocally the need for HTML structure analysis or understanding in these tasks. One such experiment could be to map all HTML tags in the web-page except the salient tags to the same token (say, ***) so that the input is now a sequence of salient tags, and ***. 
      2. There is not much novelty in the methodological aspects of the work. 

clarity,_quality,_novelty_and_reproducibility: The submission is well written and easy to understand. The three canonical tasks are described well and the adaptation of the various LLM for building models for these tasks are well explained. The proposed solution is simple and appears to be effective for the tasks considered and the datasets chosen. There is not much novelty in methodological aspects and the work is primarily empirical in nature. Experiments are designed well and should be easy to reproduce. Datasets used in the experiments have been promised to be released. The work should be interesting for practitioners.




summary_of_the_review: This work asks the question can off-the-shelf LLM trained on natural language text be used effectively for tasks that involve HTML pages. It proposes three tasks as canonical tasks in understanding HTML. It employs a variety of LLM to build models for the three tasks using a small amount of HTML data for fine tuning. It shows that LLM does help these tasks significantly. One key question not answered in this context is how much of HTML structure analysis and understanding is truly required for these questions.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper studies how pre-trained large language models (LLMs) perform on three HTML understanding tasks: (1) semantic classification of HTML elements, (2) description generation for HTML inputs and (3) autonomous web navigation of HTML pages. It found that pre-trained LLMs can work on these tasks effectively after fine-tuning. They require less task-specific data while perform better than the previous best supervised model.
strength_and_weaknesses: ## Strength
- The paper is easy to follow.
- The study of how LLMs perform on raw HTML texts is interesting and novel.

## Weaknesses
- Models that are trained on task specific data to compare against is questionable. I expect to see more task-specific models such as StructuralLM [1], MarkupLM [2] that respect the tree structure of HTML (or some other methods mentioned in the paper) to be evaluated against. However, the results seem to focus on LLMs only. I don't see the point of compare different variants of LLMs with or without pre-training. Basically, I need to see the following question answered: "For specific web tasks, do I want to use a HTML specific model trained on the small dataset or just to fine-tune LLMs"?

[1] Li, C., Bi, B., Yan, M., Wang, W., Huang, S., Huang, F. and Si, L., 2021. StructuralLM: Structural Pre-training for Form Understanding. arXiv preprint arXiv:2105.11210.
[1] Li, J., Xu, Y., Cui, L. and Wei, F., 2021. MarkupLM: Pre-training of text and markup language for visually-rich document understanding. arXiv preprint arXiv:2110.08518.
clarity,_quality,_novelty_and_reproducibility: - Clarity: The paper is easy to follow as the content of the paper is simple.
- Quality: The experiments seem to be well-done.
- Novelty: The novelty is limited in terms of methodology. It is mostly an empirical study for large LLMs on HTML texts.
- Reproducibility: Some details are missing. Either code or more detailed experimental setup should be provided in the supplementary material
summary_of_the_review: I like the idea of studying LLMs on HTML texts. However, it seems to me that the experiment setup (re. models to compare against) cannot support the main claim of why one would like to do so. Besides, the technical novelty of the paper is limited for a ML conference like ICLR; a NLP conference might be a better fit for this submission.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper investigates whether LLMs can be applied to HTML understanding to produce better-performing, more sample-efficient HTML understanding models without the need for custom NN architecture design. To achieve this goal, the authors present a suite of three benchmarking tasks for HTML understanding: (1) Semantic Classification of HTML elements, (2) Description Generation for HTML inputs, and (3) Autonomous Web Navigation of HTML pages. The authors find that a) pretraining is critical for the performance and can reduce labeled data requirements / improve sample efficiency up to 200x; b) model architecture is the second most important factor, and T5 models with bi-directional attention and encoder-decoder architecture performs the best across the board; c) given a choice, model size should be evaluated in the context of the models training and inference performance, as the model size sub-linearly predicts its performance. 
strength_and_weaknesses: Strengths:
1. The paper is well-written and easy to follow.
2. The authors conduct comprehensive evaluations and analyses over a range of architectures, dataset sizes, and baselines to make the experimental conclusion convincing.
3. The authors create and open-source a new dataset/benchmark for HTML understanding.

Weaknesses:
1. The novelty is primarily empirical, and the technical approach is essentially an application of known techniques (the standard pretrain-finetune paradigm of LLMs).
2. I examined the supplementary material provided by the authors and only found the data files. It would be better if there were codes to prove its reproducibility.
clarity,_quality,_novelty_and_reproducibility: Clarity: Most things are explained clearly either in the main paper or the appendix, except for the few points that are raised in the Weaknesses above.
Quality: well-written work.
Novelty: The novelty is primarily empirical, and the technical approach is essentially an application of known techniques (the standard pretrain-finetune paradigm of LLMs).
Reproducibility: The Supplementary Material only provides the data file. It is better to make the codes available for reproducing these results. 
summary_of_the_review: In my opinion, the audience could gain insights from the thorough empirical analysis of HTML understanding from this paper. However, the lack of technical novelty & contribution makes this work slightly below the acceptance bar of the conference. Overall, I recommend a weak reject.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes two main contributions to the direction of HTML modeling/understanding:

1. A new task and dataset, **Description Generation**, derived from CommonCrawl with 85K (HTML, element, description) tuples. 

2. Experiments across a suite of three HTML tasks (Description Generation, semantic classification from Gur et al., MiniWob web navigation), investigating the importance of pre-training corpora (natural language and/or HTML), model architecture (encoder-only, decoder-only, encoder-decoder), the scaling effect, and so on. The main finding is the effectiveness of large language models (LLM) pre-trained on natural language and fine-tuned on HTML task data, with T5-like encoder-decoder architecture being the best.
strength_and_weaknesses: Pros: The direction of HTML understanding is increasingly important, as recent work has started to show the promise of utilizing the semantics of HTML and web navigation. The paper adds solid contributions to this direction by creating a novel and interesting task, and conducting many empirical studies with insights about architecture, (sub-linear) scaling effect, importance of context window/action history, etc. 

Cons: 

- On the proposed Description Generation task, authors only spend two paragraphs presenting results, where WebD-T55-3B can obtain 90.8% accuracy (exact string match??). Does it mean the task is too easy? How to justify it from the existing semantic classification task, as in Figure 1(a) they look similar? Why in Table 3, dev numbers are much lower than test set? I feel the new task/dataset needs more explanation and justification.

- The modeling part lacks novelty, just basic pre-training + fine-tuning on LLMs, and seems there's no model design based on the features of HTML?

- On MiniWoB, it's not very surprising that using a larger model with pre-training, you can beat smaller models from scratch. Instead, the fact that LLMs are worse than RL performances should be discussed more -- does it mean the task is bad, or RL is important (but it's hard to do RL for LLM)?

- (Minor) In intro authors claim "no processing of HTML", but there is still some pre-processing?

clarity,_quality,_novelty_and_reproducibility: - Clarity: The paper is generally clear and easy to understand, some tables and figures in experiments are confusing or hard to read.

- Novelty: The new task is somewhat interesting and novel. The modeling part is not that novel.

- Reproducibility: Authors claim to open-source the Description Generation dataset.
summary_of_the_review: I believe the paper makes solid empirical contributions by introducing a new task, and studying many LLM variants on three HTML tasks, with some insights about what's important for modeling HTML. My main concern is that the new task is not clearly justified, and the modeling part might be too standard -- there is no HTML-specific model design involved, just studying basic LLMs. I think the paper can benefit from more evidence about the value of the new benchmark (e.g. what new insights/modeling capabilities it enables), or more interesting modeling.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper investigates the problem of using LLMs for understanding HTMLs. They consider three tasks: Semantic Classification of HTML elements, Description Generation for HTML inputs, and Autonomous Web Navigation of HTML pages and show encouraging results.

The paper has been discussed thoroughly during the discussion phase, and if my understanding is correct, several reviewers increased their evaluations after the authors’ response. However, the final consensus is that the paper is still below the bar for the following reasons (the scores are 5/5/5/6):
* Technical novelty and contributions are limited for conferences like ICLR.
* It lacks an in-depth comparison with HTML-specific pre-trained models, and it is not very clear how much structural information is needed for the tasks used in this paper
* It seems that the paper has room for improvement in terms of clarity and reproducibility.

Based on the above points, I can’t recommend acceptance in the current form.

justification_for_why_not_higher_score: The paper has been discussed thoroughly during the response period, and most reviewers think the paper is not strong enough for the ICLR standards yet.
Decision: Reject