Title: WEBLINX: Real-World Website Navigation with Multi-Turn Dialogue
Benchmark: WEBLINX
Summary: WebLinx introduces a large-scale benchmark of 100K interactions across 2300 expert demonstrations of conversational web navigation. Authors develop a multimodal agent able of interpreting both visual and textual input to complete web-based tasks with long context understanding and planning capabilities.
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: Multimodal reasoning, web navigation
Phenomenon defined by authors: Yes
Phenomenon definition: “We define the real-world problem of conversational web navigation: given the initial user instruction, an agent must complete a real-world task inside a web browser while communicating with the user via multi-turn dialogue.” (Page 1, Introduction)
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: No
Phenomenon contested: Contested
Task definition: The task is defined as conversational web navigation, where an agent must complete a user-specified goal on a real-world website by interacting with the web interface (e.g., clicking, typing, submitting forms) while engaging in multi-turn dialogue with the user. The agent receives inputs such as browser screenshots, DOM elements, and dialogue history to predict the next action at each turn.
Task item definition: Each item is one step in a task, where the agent sees the current web page, past actions, and what the user said, and must decide what to do next—like clicking a button or typing text. Many of these steps together make up a full task.
Task definition detail: Each task unfolds as a multi-turn dialogue between a user (called in the paper instructor) and an agent (called in the paper navigator), with actions done in a real browser environment. The task goal may not be fully known at the start and often evolves over the conversation, making long-term memory and contextual understanding important.
Task source: Real task examples (e.g. GitHub issues), Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Expert-crafted task examples (e.g. hand-written examples), Procedurally-generated task examples (e.g. Creating instances from a template)
Task ecology: Complete real task (e.g. providing medical advice to real people interactively)
Task ecology detail: Benchmark is based on interacting with people, where the end can have different results, depending on how the conversation goes - therefore making it a bit of grey area.
Dataset size: "100K interactions across 2300 expert demonstrations of conversational web navigation."
Dataset size detail: It is quite confusing from the text to count the size of the set, especially that it is based on interactions which are dependent variable - I am not fully sure if the sizes in task_dataset_size_extra are correct, I based them on table 8 as it is the only table showing active turns.
Dataset size notes: Train - 24418, validation - 1717
Dataset metadata available: Yes
Dataset metadata detail: website category, subcategory, geographic region, instructor visual access, AI assistance
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test, Train, Validation
Response format: Short free response (e.g. single word or number), Extended interaction (e.g. conversation, calling an API and processing the response), Structured response (e.g. valid JSON, API call alone)
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric definition detail: The exact match metric is calculated for each turn in the task, comparing the predicted action (such as click, say, text input) with the ground truth action. 
Metric aggregation: Simple Mean, aggregated using the micro-average of turn-level scores
Metric subscores: Yes
Metric subscores detail: Subscores are provided for element-based actions, text-based actions, and intent matching (whether the correct action type is predicted).
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No
Results realism: The benchmark is itself realistic
Human baseline: No
Authors discuss validity: No
Taxonomy: Agents / Web
New taxonomy: General Application / Web Agents