Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors propose a new web interaction benchmark called WebShop, where the agent is requested to the navigate e-commerce web-page to buy an targeted items grounded with natural language formed instruction. To this end, they collect a number of real-world products, crowd-sourced text instructions, and human trajectories. They build a baseline using text instructions and human trajectories with imitating learning and reinforcement learning, and reveal that the proposed task has much room for improvement. Transfering sim-to-real minimal loss of generalibility inditates the potential usage of the proposed benchmark.
strengths_and_weaknesses: [Strengths]
- Well written paper
- Tacking important research topic - web navigation
- Release of high quality benchmark

[Weaknesses]
- Instruction they construct may have somewhat unrealistic assumptions: 1) all attributes are known to be user before searching, 2) preferences are defined before searching the items, and 3) even those conditions are not changed during browsing the web shop. 
questions: Rather than aware of all attributes and having fixed preferences for the item, partial and incremental aware of attributes and having flexible preferences seem more realistic scenario. Several iterations of QA sessions for narrowing down the user preferences before purchasing, like in the task-oriented dialog (e.g. MultiWOZ), make the benchmark more realistic, and it would bring a great chance to the community.
limitations: Including ablation study on the effect of using image feature would be needed.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper focuses on the problem of language-grounded web interaction. The authors developed a simulated e-commerce website environment, i.e. WebShop. Different from previous work e.g. WoB, WebShop is more challenging with realistic linguistic elements. In order to better train an agent which can interact with WebShop, 1600 human trajectories are collected. Based on these human trajectories, reinforcement learning and imitation learning methods are adopted to train the agent. The experiments show that the trained agent performs better than a simple rule-based baseline, but far behind human performance. 
strengths_and_weaknesses: (1) Strengths
* The paper is generally well written.
* Both simulation evaluation and sim-to-real transfer are conducted to validate the effectiveness of the proposed framework.

(2) Weaknesses
* It seems that WebShop focuses on search-based websites. I'm wondering whether it can be extended to more general websites. For example, whether the proposed actions (search, choose) are enough for general websites? If some website needs text input, how can we obtain these web pages for building simulation environments?
* Lack of some ablations in experiments. For example, both image and text information are used in the model. What's the effect if image Information is removed?
questions: * Whether vision-language pre-trained models, e.g. CLIP, can benefit the proposed model?  
limitations: The authors have pointed out some limitations about their work.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: - The authors collect and release a dataset, WebShop, designed to train and evaluate agents in a web-based interaction task. The dataset contains natural language requests for products that match various attributes, options, and price thresholds, and subsequent human trajectories when navigating an interactive environment in which one can search, click through product pages, explore item details, look at item images, and ultimately purchase a product. 
- The authors propose two baselines: 1) a rule-based one, in which the request is fed to the search engine and the first product is picked; 2) a suite of models that uses BART to generate search queries, and ResNet and Transformer models to processes item images and text in order to predict actions and purchases. The latter method is trained using both imitation learning (w.r.t. Human trajectories) and reinforcement learning. The authors find that the proposed method outperforms the rule-based baseline, but underperforms humans by quite a large margin. Finally, the authors analyze where the gaps between human and model performance lie, and suggest various modeling improvements.

***EDIT AFTER REBUTTAL PERIOD*** 
Revising my score to 7; see below comments
strengths_and_weaknesses: ### Originality
- **Strengths**: The authors propose a novel benchmark that can be helpful to the community - indeed the closest related work (as noted) is WebGPT, though this task differs in a few ways (e.g., including images). It is clear how the work differs from other contributions.

### Quality
- **Strengths**: Claims are well supported by empirical results; the authors clearly enumerate the axes on which scores differ across methods and humans, and from this can conclude various ways in which to improve the methods. The proposed model, while simply an amalgamation of prior-used techniques, is a nice starting point for the benchmark. 
- **Weaknesses**: The authors, in my opinion, missed a few key baselines, and perhaps overstate the performance of their proposed method. While the contribution of the paper is not the method itself, but rather the dataset, I don’t think comparing it to the rule-based baseline is a proper comparison (as it essentially compares to just a BM25 retrieval performance vs. three neural network models). I also don’t fully understand why the authors omitted a fully-trained IL baseline, as they only used 1/10th of the dataset to train the IL model. Additionally, some other design choices are not quite justified empirically (e.g., using sampling instead of true beam-search)

### Clarity
- **Strengths**: Claims are well supported by empirical results; the authors clearly enumerate the axes on which scores differ across methods and humans, and from this can conclude various ways in which to improve the methods. The proposed model, while simply an amalgamation of prior-used techniques, is a nice starting point for the benchmark. 
- **Weaknesses**: The authors, in my opinion, missed a few key baselines, and perhaps overstate the performance of their proposed method. While the contribution of the paper is not the method itself, but rather the dataset, I don’t think comparing it to the rule-based baseline is a proper comparison (as it essentially compares to just a BM25 retrieval performance vs. an entire neural network). I also don’t fully understand why the authors omitted a fully-trained IL baseline, as they only used 1/10th of the dataset to train the IL model. Additionally, some other design choices are not quite justified empirically (e.g., using sampling instead of true beam-search)


### Significance
- **Strengths**: The public release of such a dataset is indeed important for those researching interactive web-based agents; if the presence of related work is any indication, this is an important direction for many. The authors also provide several “low-hanging” improvements to the model based on analyses of their baseline.
- **Weaknesses**: Human performance on the task is, in my opinion, quite low, and brings into question the validity of the dataset itself - how useful can it be if humans cannot even reach 60% task performance? This, to me, is the greatest weakness of the work.

questions: - Did you analyze why human performance is so low? What is preventing humans from finding the correct products? Or is this simply the nature of human purchase behavior?
- What is LP? It is mentioned in section 5.1 but is not properly introduced (e.g., “IL (w/o LP Choice)”)
- Was a model with full imitation learning trained? If not, why not?

limitations: The authors adequately address the limitations in the conclusion.
review_title: null

Meta-review: metareview: This paper proposes a new real-world natural language web interaction benchmark for the shopping domain and associated dataset, WebShop. The paper includes a rule-based baseline, as well as a model-based one that is trained using imitation and reinforcement learning. Performance is reported using these models in addition to human performance, showing a large gap and opportunity for better models. Reviewers made a good set of suggestions, majority of which have been covered by the authors in their rebuttal. These include additional experimentation requested by the reviewers as well as improvements to the paper presentation.
confidence: Less certain
Decision: Accept