Title: WebShop: Towards Scalable Real-World Web
Interaction with Grounded Language Agents
Benchmark: WebShop
Summary: The paper introduces WebShop, a simulated online shopping environment where agents try to follow natural language instructions to find and buy the right products. WebShop benchmark is designed to test how well agents can search, navigate, and make decisions on the web. The authors train models using imitation and reinforcement learning, and show that the best ones can even handle similar tasks on real sites like Amazon and eBay.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Natural language understanding and sequential decision-making in web environments.
Phenomenon defined by authors: No
Phenomenon definition: To evaluate agents that can understand human-provided natural language instructions and perform grounded actions in a realistic web environment, e.g generating search queries, navigating results, selecting options, and (at the end, if succesful) purchasing a product that matches the instruction.
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: No
Phenomenon contested: Contested
Task definition: The task is to follow a natural language instruction to find and purchase a product in a simulated ecommerce environment. Agent must search, navigate pages, select product options, and choose the best match based on the instruction.
Task item definition: Natural language instruction - specifying a desired product (including attributes, options, and price constraints), with the starting state of the simulated shopping environment. The agent must then complete the task by navigating and interacting with the website to find and purchase a suitable product.
Task source: Real task examples (e.g. GitHub issues), Crowd-sourced task examples (e.g. Prolific-created tasks)
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Task ecology detail: WebShop simulates online shopping using real product data and realistic ux, but it operates in a custom environment with a simplified interface and deterministic search engine. So while the core interactions reflect a real-world activity, it doesn’t capture the full complexity or variability of actual web browsing with human properly in the loop or user's behaviour.
Dataset size: 500
Dataset size notes: “[...] a total of 12,087 instructions into an i.i.d. distributed train / development / test split of 10,587 / 1,000 / 500 instances"
Dataset metadata available: Yes
Dataset metadata detail: product category, product attributes, product options, product price
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Response format: Multiple choice, Free response (e.g. summary paragarph), Extended interaction (e.g. conversation, calling an API and processing the response)
Metric definition: reward is computed based on the final product chosen by the agent, compared against known attributes, options, and price of the target product.
Metric aggregation: Simple Mean
Metric statistics: The authors report average task score and success rate across trials. They also include standard deviation/error bars in some result plots (e.g. Figure 4), mainly to show the variation across multiple runs.
Metric subscores: Yes
Metric subscores detail: Paper reports breakdowns by reward components: attribute match score, option match score, price match, and type match.
Metric access: Outputs alone
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: Yes
Human baseline: Yes
Authors discuss validity: Yes
Author validity detail: They discuss the performance gap between models and humans, quite detailed analysis of error types (e.g. failure in option matching or limited exploration), evidence of sim-to-real transfer to Amazon and eBay, aiming to indicate the external validity, as well as component-wise ablations and choice oracle (the model doesn't have to chose) experiments to diagnose bottlenecks
Taxonomy: Agents / Web
New taxonomy: General Application / Web Agents
Other notes: Here the evaluation is fully automated, which allows for easier reproduction - which seems like a significant advantage compared to others.