Web-CogReasoner: Towards Multimodal Knowledge-Induced Cognitive Reasoning for Web Agents
Authors: Yuhan Guo
Guocong
Aiwen Sun
Hongliang He
Xinyu Yang
Yue Lu
Yingji Zhang
Xuntao Guo
Dong Zhang
Jianzhuang Liu
Jiang Duan
Yijia Xiao
Liangjian Wen
Haiming Xu
Yong Dai
Venue: ICLR 2026 Poster (2026)
Keywords: Web agent
Web UI
Cognitive Reasoning
Web knowledge
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Multimodal large-scale models have significantly advanced the development of web agents, enabling them to perceive and interact with the digital environment in a manner analogous to human cognition. In this paper, we argue that web agents must first acquire sufficient knowledge to engage in cognitive reasoning effectively. Therefore, we decompose a web agent's capabilities into two essential stages: knowledge content learning and cognitive processes. To formalize this, we propose Web-CogKnowledge Framework, which categorizes knowledge into Factual, Conceptual, and Procedural domains. In this framework, knowledge content learning corresponds to the agent's processes of Memorizing and Understanding, which rely on the former two types of knowledge, respectively, representing the "what" of learning. Conversely, cognitive processes correspond to Exploring, grounded in Procedural knowledge, defining the "how" of reasoning and action. To facilitate knowledge acquisition, we construct the Web-CogDataset, a structured resource curated from 14 real-world websites, designed to instill the core knowledge necessary for a web agent systematically. This dataset serves as the agent's conceptual grounding—the "nouns" upon which comprehension is built—as well as the basis for learning how to reason and act. Building on this foundation, we operationalize these processes through a novel knowledge-driven Chain-of-Thought (CoT) reasoning framework, developing and training our proposed multimodal web agent, the Web-CogReasoner. Extensive experimentation reveals its significant superiority over existing models, particularly in its capacity for generalization to unseen tasks where its structured knowledge proves decisive. To facilitate rigorous and systematic evaluation, we introduce the Web-CogBench, a comprehensive evaluation suite designed to assess and compare agent performance across the delineated knowledge domains and cognitive capabilities. Our code and data are open sourced at https://github.com/Gnonymous/Web-CogReasoner.

Published as a conference paper at ICLR 2026
WEB-COGREASONER:
TOWARDS MULTIMODAL KNOWLEDGE-INDUCED COGNITIVE REASONING FOR WEB AGENTS
Yuhan Guol.21, Cong Guo Aiwen Sun", Hongliang He,, Xinyu Yang", Yue Lu', Yingji Zhang", Xuntao Guo Dong Zhang'. Jianzhuang Liu Jiang Duan 1.12
Yijia Xiao Liangjinn Wen Hai-Ming Xu Yong Dai
SSotthwestern University of Finance and Economics, SShanhaii Jiao Tong University, 3Central South University, "Hithink Research, 'Westlake University.
"Harbin Institute of Technology, 'University of Manchester
8 University California, Los Angeles, Uniivesiity of Adelaide, Fudan University,
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Chhnnguu Everimaging Science and Technology Co., Ltd.
httts:://Goonmoout withub. Atw.auubbooeebe-ooooasooseoe
Web-CogDataset & Web-CogBench
Stagel Memorizing
81
Stage3 Exploring
Element Attribute Recognition
Sub-elements Prediction
Fectual Knowledge
Page Change Prediction Next Page Prediction
Popup Close
Single Step Exploration User's Intention Prediction
Procedural
Noisy Multi-Step Web Task
Stage wio World
Web-Cog
Stage Now can explore the world!
Knowledge
Stops Try to understend
Conceptud knowledge
Coption & Q.A
Element Understanding WebPage Understanding
Evolving into Web- CogReasoner
Stage2 Understanding
62*
Impact Knowledges
Web-CogBench
Source
$7.99
11.3
Ned
19.2
Undestandinng
WebVoyager
Multimodal- Mib
19.2
Aftiitute Recegetic
Zntent
Shop
-10 1alls 18-
-Dorn V1.
Popup
Figure 1: Visualization of the Web- Leab-CogKnowledgee Framework along with the experimental results.
These authors contributed equally this work. Project Leader. Cooressponing authors.

---

Published as a conference paper at ICLR 2026
ABSTRACT
Multimodal large-scale models have significantly advanced the development of web agents, enabling them perceive and interact with the digital environment in a manner analogous to human cognition. In this paper, we argue that web agents must first acquire sufficient knowledge to effectively engage in cognitive reasoning" Therefore, we decompose a web agent's 's capabilities into two essential [
phases: knowledge content learning and cognitive processes. To formalize this, Framework, which categorizes knowledge into Factual, Conceptual, and Procedural domains. this framework, knowledge content learning corresponds to the agent's processes of Memorizing and Under- standing, which rely on the first two types of knowledge respectively, representing the "what" of learning. Conversely, cognitive processes correspond to Exploring, grounded in Procedural knowledge, defining the "how" of reasoning and action. To facilitate knowledge acquisition, we construct Web-CogDataset, structured resource curated from 14 real-world websites, designed to systematically instill the core knowledge necessary for a web agent. This dataset serves as the agent's s conceptual grounding-tth the "nouns" upon which comprehension is built-as as well as the basis for learning how to reason and act. Building on this foundation, we op- Sihin-of-Thoulle
we propose the Web-CogKnowledee
erationalize these processes through a novel knowledge-driven (CoT) reasoning framework, developing and training our proposed multimodal web agent, Web-CogReasone superiority over existing models, particularly its capacity for generalization to unseen tasks where its structured knowledge proves decisive. To facilitate rigorous and systematic evaluation, we introduce Web-CogBench, a comprehensive eval- uation suite designed to assess and compare agent performance across the delin- eated knowledge domains and cognitive capabilities. Our code and data are open sourced at at //tp::::://gtttttt wtt.:/com/mmmmooeeeeeeeee Webb CogeeeoeeCooooeeeoeeeeeeee
Extensive experimentation reveals its significant
INTRODUCTION
The advent of large-scale models marks a milestone in artificial intelligence, with Large Multimodal Models (LMMs) greatly expanding application horizons. Al agents have become the primary vehicle for deploying these models, enabling capabilities in code generation (Hui et al. 2024 Jiang et al.
2024), image and video synthesis (Huang et al.) 2025 Bie et al., 20244]
Assran et al. 2025 Liu
et al. 2024c), and academic research (Li et al. 2025 Zhang al.
2025 Recent progress has also
highlighted the growing importance of web agents.
Web agents have evolved from early rule-based systems to modern approaches leveraging Large Language Models (LLMs) and Language Vision Models (LVMs) (Wang et al., 2024 Ning et al., 2025 /Zhan al.,, 2024 Sapkota al. 2025). LLM-powered agents typically convert HTML. or Accessibility Tree inputs into natural language prompts for reasoning and action. With LVMs, agents gain perceptual abilities akin to human vision, allowing them process multimodal content on web pages. Broadly, web agents can be categorized as: (1) text-only (Zhou et al.) [023] Li et al., 2023), which miss visual cues: (2) vision-only (Qin al.] (025), which lack structured data; and (3) hybrid (Koh et al. 2024 He et al. 2024b), which integrate both modalities.
LLMs and LVMs pre-trained on general-domale knowledge provide strong foundations but remain limited in specialized tasks, creating performance bottlenecks. Prior mowleedge-enhnceeee ods often lack systematic or theoretical grounding. To address this, we draw inspiration from Bloom's s Taxonomy (Ormell 1974 Conklin 2005), which divides learning into two phases; Knowl- edge Content Learning and Cognitive Processes.
meth-
In our paradigm, the first phase builds a multi-layered foundation: Factual Knowledge, covering basic concepts, and Conceptual Knowledge, capturing their interrelations. This equips the agent with core web knowledge and its application to familiar tasks. The second phase develops Proce- dural Knowledge, providing logical frameworks to synthesize prior knowledge for reasoning and exploration. This enables the agent to "leam how to learn." creatively leveraging self- eeffknnowedgge to
Drawing inspiration from Bloom's educational philosophy, comerstone of modemn pedagogy.
2

---

Published as a conference paper at ICLR 2026
solve novel challenges. This mirrors the human learning trajectory: first accumulate knowledge through education (Phase 1), and then based on that foundation of knowledge and experience, we learn to apply, innovate, and create (Phase 2).
To support this, we construct Web-CogDatasel from 14 prominent websites with 12 fine- anneggraneed tasks, and design knowledge-guice reasoning templates combined with imitation learning to instill the required cognitive faculties. Rigorous evaluations on public and in-domain benchmarks show that our method consistently surpasses state-of-the-art baselines, with the performance advantage es- pecially pronounced in knowledge-intensiv tasks. These results confirm that structured knowledge acquisition crucial for enabling agents excel complex, domain -epaain-peccficc scenarios.
In summary, our contributions are threefold:
1. Drawing inspiration from Bloom's taxonomy and established human educational paradigms, we propose the Web-CogKnowledge Framework, a systematic, two-phase train- ing methodology designed to enhance the cognitive capabilities of web agents. As shown in Figure built upon this framework, we develop Web-Cogte Web--CogReaooner. Rigorous bench- marking demonstrates that agents trained under our framework achieve a significant per- formance improvement over current state-of-the-an models.
2. We construct Wetsee, structured curriculum consisting of 12 fine-grained and progressively challenging tasks. These tasks are meticulously designed incrementally build the agent's web knowledge, cognitive capability, and higher-order reasoning.
3. To enable comprehensive and robust evaluation, we introduce Web-CogBench, novel benchmark specifically designed to assess whether a web agent possesses the requisite prior knowledge and cognitive capabilities for effective web navigation. This benchmark will be released publicly to foster further research in this area.
RELATED WORK
2.1
WEB AGENT
Early work on web understanding focused structured HTML, addressing tasks like semantic clas- sification, description generation, and navigation (Gur et al. 2022). with AutoWebGIMM (Lai et al., 2024) further applying curriculum learning for structure recognition, component understanding. and progressively complex task execution. More recent studies leverage visual signals: SeeClick (Cheng
al. 2024) linked elements to textual descriptions to enhance localization, CogAgent (Hong al., 2024 combined high-resolution cross- module modeling with large GUI dataset for VQA and navigation, OmniParser (Wan et al. 2024) unified text spotting, extraction, and table recognition, UI-TARS (Qin et al.) 2025) directly maps screenshots to actions, and UGround (Gou et al., (024) trained on 1OM GUI elements for robust desktop and mobile performance. Multimodal approaches integrate text and vision: Web Voyager (He et al. (2024a) combines screenshots with bounding boxes and accessibility trees, SeeAct (Zheng et al.) 2024) grounds text plans via GPT-4V, and TAG (Xu et al. 2025) exploits pretrained attention for grounding without fine-tuning.
2.2 WEB AGENT EVALUATION
Benchmarks are categorized into browsing and understanding. For browsing, offline datasets like
Mind2Web (Deng et al. 2023).
Mullimodaal-Mind2Weeh AutoWebBench (Lai et al.) [024), and 2024) test multi-step task execution, while online environments such as
WebVLN-vI (Chen et al.
Mini-WoB (Liu et al.;
2018) Webshop (Yao et al. 2022). and WebArena (Zhou et al. 2023)
allow real-time evaluation. Mini-WoBre emphasizes low-level UI operations, whereas Webshop
and WebArena simulate complex tasks. VisualWebArena (Koh et al. 2024) further adds multimodal inputs for dynamic interactions.
For web understanding, WEBQA (Chang et al. 2022) evaluates open-domain multi-hop reasoning. ScreenQA (Hsiao et al. 2022) and ScreenAl (Baechler et al., (024) focus on screen comprehen- sion, with ScreenQA targeting UI recognition and contextual QA, while ScreenAI extends this into three subtasks: Screen Annotation, ScreenQA Short, and Complex ScreenQA. Together, they assess layout understanding, semantic interpretation, and reasoning in visually dense interfaces.
3

---

Published as a conference paper at ICLR 2026
3 WEB-COGKKNOWLEDGE FRAMEWORK
3.1 BLOOM'S TAXONOMY
The Bloom's Taxonomy (Anderson & Krathwohl, 2001) presents a two-dimensional framework: knowledge content learning and cognitive processes, embodying a "shallow-to-deep" instructional methodology. It structures learning from foundational facts and concepts to complex procedures, ensuring a solid knowledge base before higher-level reasoning.
This progression can be formalized through four hierarchical types of knowledge: Factual, Concep- tual, and Procedural Knowledge. Further details are in Appendix A.1.
Web-CogDataset
Knnowledge Improving agent's dbility accurately identify facts elements webpages
Concaptual Web knowledge:
Enhancing agent's 's ability to aralyze the deep semontics and contextul relationships within web content.
Procedural Web Knowledge: Developing agent' 's ability to accomplish complex and real-world web tasks.
&
thedeeent antaurod bau keee
Quiees Tull neaku
Gandon Cheacs tumetooe
Search destinatio..
Addiqueets
dbuf Sileet
@ (3 Lal SleAd
Arctis
besign
Priceshncuuie
Talle
fovorite
Quest fovorite
Med Pe Wethoa
oepeiiel wer offer cirleg nower whee the fol
Seppming Nurvary
Syrig" olaget
Dhngeity
Bura,
Qualdo +5,0
Preding, Adattia buitt
Batran-Lübchin..
Vittorio Venet.. coometere aunyy May 11-18
7,000 iolometers mwwy May 8-133
Du Guact "2..
The web faturcal
cliing
May
May
V24,004
44,000 wiregs
"tp..
Qustoa Defemmte wichh UTVL
Y4,781 ins
V222011 go-i
anafic?
show fncto
krting
Gunst fovprite
Metinarlge jen Po moniet
docrrpiin entche
Accessibla.
the Purhepeet demen.
Deleng
[PHoottttite Aehwel boothe
Sitlleeri froaed The
Mrcko Thomen Sang
1940 way!
(H
A
Secaso deeeet
Quliu Toun Moded di The
weatts screnmeet recllk plp
tul All parfit eiperdlc
Feed Aow
Roringe carter Pluvaaan
iramt plas of The park ediak schie.
AnSrope Thcaing ampricciaion
dipleyy Notron tualdn proades growery mel Alled averts cicerndinonnetteo
Armen Unedly eitooinoee anrund ine.
Anmarr
Menngrgl Loyolt Decrofiee
fuct unily dever ganen
Summayy
Feel Aclis Fedeamore...
Figure 2: Iluutrationn of Web-CogDataset.
3.2 WEB-COGKNOWLEEGGE
Motivated by Bloom's 's taxonomy, we propose a hicrarchical web knowledge framework that struc- tures web knowledge according its taxonomy, collects corresponding knowledge each level, and trains the model correspondingly. We refer to this knowledge as Web-CogKnooledge. This Web-CogKnowledge decomposes knowledge into three levels:

---

Published as a conference paper at ICLR 2026
1. Factual knowledge: concrete information extracted from web contents, such identifying the attributes individual web elements and predicting the immediate, direct consequences of single interaction.
2. Conceptual knowledge: semantic relationships and abstract patterns underlying webpage content and structures, such as inferring the function of interface components, compre- hending the overall purpose and structure of webpage, and interpreting its multimodal content.
3. Procedural knowledge: actionable know-how for accomplishing specific tasks through interaction, including planning. decision-makee and sequential execution. Examples in- clude executing goal-oriented action sequences, inferring user intent from observed behav- iors, and handling unexpected interruptions to complete complex tasks.
This taxonomy aligns each knowledge type with a corresponding cognitive competency required for web-based reasoning and interaction.
To fully leverage the potential of Neb-CogKnowledge. leading to the ultimate realization of the WebbCogKnowleedge Framework, we first collect multimodal metadata from 14 real-worle web- sites (see Figure 6) and then design Web-CogDataset with diverse sets of web tasks for web-agent training Section 3.33 and finally construct Web-CogRench Section (3.4] built from meticulously chosen subsets. Specifically, we curate the Web-CogReasoner's learning and training process in Section due to its importance and complexity. More details about the data collection and cleaning process can be found in Appendix B.1
3.3 WEB-COGDATASER
By selectively crawling metadata from 14 representative websites and aligning it with the hierar- chical design of Web-CogKnowledge,, we construct Web-CogDataset, large-scale, hierarchically structured dataset tailored for knowledge-centee web reasoning. The dataset spans three layers of knowledge: Factual, Conceptual, and Procedural. Each is mapped to carefully designed task families that progressively cultivate perception, comprehension, and action-oriented reasoning.
As illustrated in Figure 2 these tasks together form a coherent pipeline that transitions agents from identifying elemental attributes, grasping semantic patterns and page structures, and finally to executing complex, goal-directed interactions under realistic constraints. This organization mirrors human learning trajectories, ensuring that higher-order reasoning is built on solid perceptual and conceptual foundations.
Detailed task definitions, implementation protocols, and examples are provided in Appendix C.I Dataset statistics are reported in Table [3 To ensure data quality and diversity, we further analyze the dataset composition and annotation reliability in Appendix C.T
3.4 WEB-COGBENCH
To evaluate the cognitive capabilities enabled by our knowledge-centric framework, we introduce Web-CogBench. While our training dataset organized by knowledge type (Factual, Concep- tual, Procedural). Web-CogBench measures agent performance across three corresponding abili- ties: Memorizing, Understanding, and Exploring. Curated from representative subset of Web- CogDataset, it assesses how effectively agent applies learned knowledge in complex web texts. Detailed statistics are in Table [1] and the evaluation dimensions align with our hierarchical knowledge framework. A complete definition of all tasks is provided in the appendix C.I.2
Memorizing Assessing the agent's ability to recall and recognize concrete information, directly corresponding to the acquisition of Factual Knowledge. It evaluates whether the agent can accurately identify the attributes of web elements and the state of webpage.
Understanding Measuring the agent's capacity for semantic interpretation, aligning with the mas- tery of Conceptual Knowledge. It tests whether the agent can move beyond mere identification to comprehend the function of elements and the contextual relationships within page.
5

---

Published as a conference paper at ICLR 2026
Table 1: Statistics of Web-CogBench.
Task
Cognition
Metric
#Num
Element Attribute Recognition
ROUGE-L
249
Next Page Prediction
Memorizing
Accuracy
Source Element Prediction
Aceuracy
Element Understanding
LVM Judge
200
Understanding
WebPage Understanding
LVM Judge
77
User's Intention Prediction
LVM Judge
105
Exploring
Popup Close
Accuracy
58
Single Step Exploration
Accuracy
62
Total
876
Exploring Evaluating the agent's ability to plan and execute goal-oriented actions, reflecting the application of Procedural Knowledge. It assesses whether the agent strategically navigate the web, handle interruptions, and complete multi-step tasks to fulfill user goals.
4 WEB-COGREASONER
4.1 PROBLEM SETUP
We model the interaction between Web-CogReasoner and the environment as partially observable Markov decision process (POMDP): P (S, O, T. T, R), where S denotes the webpage state, A the action space (Table I0,, O the observation space, K the internal knowledge, T the transition function, and R the reward function. At each step the agent receives a screenshot and its accessi- bility tree (AX Tree), forms reasoning thought h4, and selects an action a, under policy This process continues until task completion step limits are reached, with binary rewards indicating success failure.
4.2 FRAMEWORK OVERVIEW
Web- Reb-CogReasoner is multimodal knowledge-driver reasoning system built LVMs and trained on Web-Cogu Web-CooDattaset (Section 3.3). As illustrated in Figure 3 tackles complex web tasks by gen- erating Knowledge-driven Srain-of-Thoughe (CoT), in which each reasoning stage is explicitly grounded in layer of Web-CogKnowledge.
Knovledge-driven CoT Reasoning The core of Web-CogReasoner is a structured chain-of- thought (CoT) reasoning process (Figure B). decomposed into three layers: Factual (identify- ing page elements and states: "What is on the page?"), Concepiual (inferring roles and interac- tions: "What does it mean?"), and Procedural (planning goal-directed steps: "How accomplish the task?"). This layered reasoning maps task prompts to executable actions: Task Prompt + Knowledge-driven CoT + Plan Action. The agent initiates with Task and observes the web- page (Observation Space). This structured reasoning guides the Planning phase, which decomposes the task and formulates step-by-step strategy. The process culminates concrete Action to be executed on the webpage.
EXPERIMENT
5.1 EXPERIMENTAL SETUP
Models and Baselines Our proposed Web-CogReasoner builds on Qwen2.5-VL-7B (Bai al. 2025), extended with knowledge-induced reasoning framework. Training details are provided in Appendix D.I]] For comparison, include diverse baselines: the vanilla Qwen2.5-VI.-7B

---

Published as a conference paper at ICLR 2026
Factual Web Krowledge
Gorceptull Web Krowledge
Procedural Web Knowledp
tha
Web-CogReasoner
#1 leaas
Shore dos to oode 900t The aee has cirasty nodgored the "hrd Store nechenet the Agp exibitte mn
Observation Space
menthmm fely oemy Apple store war coppe thr loctees at thee to rnsertiion her
Plani m Knowledge- Sagge-diien COT Recse
cotte for ronge cry tevigruaaion showe tng. the pagure slin op med aol lomeor of Formy an
Ped adecriccasre
eet say Blrwen Anayels
106 searc cosembble none Search leotti strre send""
ofher diukeg antrectio Lobit
cod dgaal storaa
will things aftee aetecceee, After kocaiie The susite dealey
tore Store Lariige
4REA
valle "Apple fors vaator Comtrrt
living lelades shore spenty tetings
crpanteod rad with oer non and
arfomnttion ond purtily dincc sia..
1RE4 sie nelpenoe attorrible nese Thee poge"
Doaatred shatt This orrr uort to ahar dar diuke insricn usere all di vou seru stone latingt camerr
bection whic Hors ferm laougg dalkeyl yel Uier une the whe oene lacoog temoo de tubl wr drn ten rey not everal
Far thore casy eccnte mnch woreecten
Trdlctory littory Nedens
the "Pade theee" section iy leper the 23P carle The legle/ is tospe the search hew
ocds escrch
Action Space
Ageu staes entorel
tee cosl oail thaee near the wis
mealss the realts iis wn
the priweyy this sark ayeetnnn the cads
Seu
lopt loglod srep is theasrch oer shance
ACGE tke chrsi deve olevan realss
soci per
NLT
50023
Figure 3: Ilstrrtinng the Knowledge-driven Chain of Thought (CoT) process.
prro/ffw-shot), Open Web Vovager (He et al., 2024b) in IL and Max variants, the open-source UI-TARS-7B-SFT (Qin et al.) (025), and two powerful commercial LVMs, Claude Sonnet 4 and Gemini 2.5 Pro. Together these cover foundational L.Mss, end-to-end web agents, and general- purpose multimodal models.
Datasets Web-CogReasoneris is trained via supervised fine-tuning on the curated Web-CogDataset (S3c.3.3), which aligns samples with Factual, Conceptual, and Procedural knowledge using screen- shots, accessibility trees, and reasoning trajectories. Evaluation is conducted on four datasets. Our custom Web-CogBench (Sec. (34) assesses cognitive dimensions of Memorizing, Understanding, and Exploring. Ben (Liu et al. 2024b) provides 1.5K curated tasks across 139 real websites, testing grounding and reasoning in diverse settings. Web WebVoyager (He et al. 2024b) con- tains 643 queries across 15 seen sites, measuring performance in familiar environments. Online (Deng et al. (023) evaluates cross-task and cross-website generalization with queries from both known and unseen domains.
5.2 MAIN RESULTS
Results on Web-CogBench We evaluate Web-CooReeasone on Web-CogBench, which assesses foundational knowledge and cognitive reasoning across twelve web tasks. As Table shows, our model outperforms both commercial and open-source baselines, owing to the integration of structured Web-CogKnowledge (factual, conceptual, procedural) with Knowledge-driven Chain-of- Thought reasoning. This combination enables accurate perception of web elements and informed, step-wise decision-making. The strong synergy between visual recognition and cognitive reason- ing allows Web-CogReassoner to excel on diverse tasks and generalize effectively to web navigation scenarios. We also provide a detailed LVM evaluation protocol and inter-rater reliability analysis in Appendix D.2 to validate the robustness of our scoring metrics.
7

---

Published as a conference paper at ICLR 2026
Table 2: Performance evaluation the Web-CogBench benchmark.
Element Attribute Rec
Next Page Pre
Source Element Pre
Element Understanding
Model
Claude Sonnet
79.7
93.5
62.5
62.8
Gemini 2.5 Pro
79.8
94.6
84.4
62.6
Qwen2.5-VL-7B UI-TARs-7B-SFT
53.2
83.9
63.6
60.0
63.5
88.0
31.3
48.0
Web-CogReasoner (Ours)
91.4
93.5
87.5
69.2
WebPage Understanding
User Intent Pre
Popup Close
Single Step Exp
Overall
Claude Sonnet 4 Gemini 2.5 Pro
54.3
64.7
100
96.8
76.8
73.5
51.9
96.6
98.4
80.2
Qwen2.5-VL-7B
62.0
51.9
91.4
90.3
69.8
UI-TARs-7B-SFT
48.0
32.4
25.9
33.9
46.4
Web-Coggeasoonr (Ours)
79.0
61.4
98.3
95.2
84.4
Table 3: Performance evaluation the VisualWebBeech benchmark.
Perception-Oriae Oriented Tasks
Perception Avg
Reasoning- Orienntt Tasks
Reasoning Avg
Overall Avg
Model
Action Ground
WehQA
HeadOCR
OCR
Ground
Prediction
Claude Sonnet 4
73.3
72.6
96.2
80.7
81.1
96.1
96.3
91.2
85.9
Gemini 2.5 Pro
74.9
70.8
95.1
80.3
91.8
96.8
90.3
93.0
86.6
Qwen2.5 VL
70.8
71.7
81.4
74.6
77.5
86.8
68.0
77.4
76.0
UI-TARs-7B. SFT
71.3
78.7
97.2
82.4
91.8
91.8
85.4
89.7
86.0
Web-CogReasouer (Ours)
86.3
Results on WebBenhe We evaluate visual understanding on VisualWebBench (Liu et al, 2024b). As Table shows, Web-CogReasoner achieves the highest average (86.3%). slightly above UI-TARS (86.0%). However, UI-TARs performs poorly on Web-CogBench (46.4%), high- lighting that strong visual perception alone does not ensure robust cognitive reasoning. In contrast, Web-CogReasone consistently excels across both visual and reasoning benchmarks, demonstrating the effective integration of precise visual perception with structured knowledge-drive reasoning a dual capability essential for reliable web agents.
Table 4: Task success rates on the WebVoyager. The "Overall" score is the average success rate.
Agent
Allrecipes
Amazon
Apple
ArXiv
GitHub
Booking
ESPN
Coursera
Claude Sonnet 4
26.7% 7%
87.8%
48.8%
69.8%
68.3%
2.3%
45.5%
83.3%
Gemini 2.5 Pro
60.0%
63.4%
62.8%
67.4%
68.3%
9.1%
56.8%
73.8%
Qwen2. Qwen2.5-VL-7B
0.0%
0.0%
0.0%
4.7%
0.0%
0.0%
0.0%
2.3%
OpenWeb
17.8%
12.2%
20.9%
14.0%
14.6%
9.1%
9.1%
31.0%
Open Voyageenan
22.2%
29.3%
32.6%
20.9%
26.8%
11.4%
11.4%
42.9%
Web-CogReasoner (Ours)
26.7% %
31.7%
32.6%
%
29.3%
2.3%
15.9%
54.8%
BBC News
Cambridge Dictionary
Google Flights
Google Map
Wolfram Alpha
Huggingface
Overall
Claude Somnet 4
23.8%
37.2%
4.8%
80.5%
48.8%
82.6%
47.7%
Gemini 2.5 Pro
52.3%
76.7%
4.8%
75.6%
58.1%
82.6%
54.9%
Dwen2.5-VL--2
0.0%.
11.6%
0.0%
2.4%
7.0%
2.2%
2.2%
OpenWebWayageen
9,5%
37.2%
9.5%
22.0%
20.9%
26.1 1%
18.1%
OpenWebVoyagere
14.3%
34.9%
21.4%
29.3%
32.6%
37.0%
26.2%
Web-CogReasoner (Ours)
14.3%
55.8%
9.5%
39.0%
37.2%
39.1
30.2%
8

---

Published as a conference paper at ICLR 2026
Table 5: Performance comparison on the Online-Mind2Web under cross- cross-task and cross-website
Cross-task (Unseen Tasks)
Cross-eb (Unseen Websites)
Agent
Entertainment
Shopping
Travel
Overall
Entertainment
Shopping
Travel
Overall
Clande Sonnet
44.9%
35.3%
40.0%
40.2%
45.5%
6.7%
14.0%
21.7%
Gemini 2.5 Pro
46.9%
35.3%
28.3%
37.5%
42.4%
10.0%
23.3%
25.5%
Open OpenWeb
22.4%
29.4%
15.2%
20.5%
3.0%
8.7%
23.3%
11.7%
Qwen2.5-VL-7B
2.2%
0.0%
0.0%
0.0%
1.0%
OpenWebNoyyggnn.
8.2%
5.9%
4.3%
6.3%
3.0%
5.8%
4,7%
6.6%
Web-CogReasoner (Ours)
16.3%
23.5% %
15.2%
17.0%
12.1%
7.7%
9.3%
10.1%
Results on Online Web Tasks We evaluate Web WeasSor live web tasks using Wey ager and Online MullmoddalMind2Weal to assess practical utility and generalization to unseen websites and multi-step tasks (UL-TARs-7B-STT Web-CogReasoner achieves state-of ofate-of-the-art results among open-souree agents, suggesting that in- tegrating structured Web-CogKnowleeee with Chain-of-Thought reasoning supports more reliable perception and execution. For Mind2Web, Open Web VoyagerMax is not strictly comparable zero- shot baseline due additional sample collection and retraining on high-error sites; nevertheless, without task-specific fine-tuning, Web-CogReasoner outperforms Open VoyagerLL and stays competitive with Open Web Web Voyagerrgrr highlighting robust generalization.
omitted due to missing online inference scripts).
Table 6: Average steps per successful task across different benchmarks.
Mind2Web Cross- -assk
Mind2Weh Cross- Web
Agent
Web Voyager
Final Avg
Claude
7.35
10.89
11.04
9.76
Gemini
6.68
7.74
10.30
8.24
Open Web VoyagerMa
5.07
7.59
6.91
6.52
Qwen2.5-VLL VL-77B
7.69
12.00
13.00
10.9
OpenWeb Web Voyagern.
5.26
7.00
9.29
7.18
Ours
4.73
7.37
8.89
7.00
Results Average Steps reports the average number of steps for successful online tasks.
Our approach consistently achieves high efficiency, particularly in cross-domale scenarios, indi- cating that the model effectively balances streamlined task execution with robust generalization to unseen environments.
5.3 ABLATION STUDY
To empirically validate the effectiveness of our Web-CogKnowledge Framework, we conduct two- fold ablation study. First, we evaluate the cumulative gains of our curriculum learning strategy (Table 77) Second, to address specific inquiries regarding the necessity of each knowledge layer and the reasoning mechanism, we provide a detailed component analysis on both Web-CogBench and Web Voyager (Tables and 9
Table 7: Cumulative Gains: Impact of progressive knowledge training Web-CogBench
Model Configuration
Memorizing
Understanding
Exploring
Overall
Swen2.5-VL-7B (Basc Model)
67.6
61.0
77.9
69.8
Factual Knowledge (S1)
85.5
(+17.9)
64.2
60.1
72.1
Conceptual Knowledge (S2)
88.1
75.5 (+11.3)
65.8
78.3
Procedural Knowledge (S3)
90.8
74.1
85.0 (+19.2)
84.4
Cumulative Impact of Curriculum Learning We ablate the curriculum components by incre- mentally adding Factual, Conceptual, and Procedural knowledge to the base Qwen2.5-VL. VL- and evaluating Web-CogBench. Factual knowledge improves perceptual grounding for element

---

Published as a conference paper at ICLR 2026
recognition (Memorizing), Conceptual knowledge adds semantic/functional understanding (Under-
standing), and Procedural knowledge enables goal-directed planning and execution (Exploring). Ad- ditional qualitative examples provided in Appendix E
Table 8: Component Analysis Web-CogBench: Validating hierarchical dependency,
Model
Memorizing
Understanding
Exploring
Overall
Base model
67.6
61.0
77.9
69.81
S1 only
85.5
64.2
60.1
72.12
S2 only
59.88
68.03
60.00
61.96
S3 only
52.82
46.40
78.00
60.66
S1+S2
88.1
75.5
65.8
78.33
S14S3
85.11
53.53
82.31
76.17
S2+S3
64.87
69.74
81.41
72.29
S1+S2+S3 (Full)
90.8
74.1
85.0
84.44
Table 9: Impact of Knowledge & KCoT on Web WebVoyager, real-world online tasks.
Model
Amazon
Cambridge
Coursera
GitHub
Overall
SI only
12. 19%
25.58%
14.28%
7.14%
12.67%
S3 only
17.07%
11.62%
16.66%
14.28%
13.14%
S1+S3
29.26%
34.88%
28.57%
16.66%
23.47%
S1+S2+S3 (w/o KCoT)
19.51%
51.16%
26.19%
23.80%
25.35%
S1+52+S3 (w/ KCoT)
31.7%
55.8%
54.8%
29.3%
42.9%
Hierarchical Dependency of Knowledge To verify that curriculum stages are hierarchically dependent layers, we analyze the component ablations in Table g and the online results in Table
Low-level Knowledge is Prerequisite: In Table single-stage models (S2 only, S3 only) underperform on overall metrics. Adding Factual training (S1) substantially strengthens higher stages; for instance, on WebVoyager (Table 9 S1+S3 nearly doubles the success rate vs. S3 only (23.47% vs. 13.14%), indicating that procedural exploration relies on accurate factual grounding.
Integration is Critical:
Although individual stages specialize (e.g., S3 benefits Exploring), only the integrated model (S1+S2+S3) achieves consistently strong performance across dimensions, suggesting that effective web agents require the full cognitive stack.
Reasoning Activation via KCoT Finally, investigate the role of reasoning framework. While the full combination of data (S1+S2+S3) builds a strong latent representation, explicit reason- ing is required to utilize it. As shown in Table removing the Knowledge-driven Chain-of-Thourr (w/o KCoT) causes a sharp drop online success rate from 42.9% % to 25.35 %. This indicates that KCoT acts as a crucial activator, bridging the gap between possessing knowledge and applying dynamically for eecisiinnmmaking
CONCLUSION
We present Web- Reb-CogReas,ner, multimodal cognitive-inspired framework for web agents that systematically instills Factual, Conceptual, and Procedural Knowledge, following Bloom's Tax- onomy. By coupling the Web-CogKnowledge Framework with the Web-CogDataset and Web- CogBench, our approach enables interpretable, step-wise reasoning through knowledge-driven Chain-of- ThhiuhhtT yielding strong performance on complex web navigation and instruction- following tasks. Ablation and qualitative analyses confirm the indispensability of each knowledge stage, demonstrating how a structured curriculum produces robust perceptual and cognitive capabil- ities. While current results rely imitation learning, future work aims to integrate reinforcement learning to enhance exploration, generalization, and autonomous discovery of procedural knowl- edge, advancing toward truly adaptive and self-directed web agents.
10

---

Published as a conference paper at ICLR 2026
7 ACKNOWLEDGMENTS
This work was supported by the Major Science and Technology Special Project of the Sichuan Provincial Department of Science and Technology (Grant No. 2024ZIZX0002), the Sichuan Provincial Innovation Group Project (Grant No. 2024NSFTD0054), Fundamental Research Funds for the Central Universities (JBK202510011..
ETHICS STATEMENT
This work does not involve human subjects, private or sensitive data, or any personally identifiable information. All datasets employed in our experiments are publicly available and have been used accordance with their respective licenses. We acknowledge the potential societal risks associated with large language models, including issues of fairness, bias, and misuse. While these concerns are the primary focus of this work, we have taken steps to ensure responsible experimentation, cluding transparent reporting of datasets, models, and evaluation protocols. The release of our code and models will be accompanied by appropriate documentation and usage guidelines mitigate unintended applications.
REPRODUCIBILITY STATEMENT
We have made every effort to ensure that our results fully reproducible. All relevant de- tails inss-niiniinng the proposed methodology, model architecture, training objectives, evaluation pro- tocols, hyperparameter settings, ablation configurations, and data preprocessing pipeline-ane thor- oughly documented in the appendix. All datasets used in our experiments are publicly accessible, experiments
and instructions for reproducing pendent verification and extension of our work, we will release the source code, trained model checkpoints, and experiment scripts the near future.
clearly provided. To further support inde-
10
LLM USAGE
Large Language Models (LLMs) were used to aid in the writing and polishing of the manuscript. Specifically, we used an LLM to assist in refining the language, improving readability, and ensuring clarity in various sections of the paper, The model helped with tasks such sentence rephrasing, grammar checking, and enhancing the overall flow of the text.
In our research, LLMs were used to assist in generating training tasks. Specifically, Section 13 we used the LLM infer the functions of web page elements based on web page screenshots and structured text. then used these results to build training data, which helped improve the target model's 's understanding and prediction capabilities.
is important note that the LLM was not involved in developing the research concepts, designing the research methodology, or formulating the experimental protocols. All research concepts, ideas, and analyses were independently developed and implemented by the authors. The authors bear full responsibility for the content of the manuscript, including any text generated polished by the LLM. We have ensured that the text generated by the LLM complies with ethical guidelines and does not involve plagiarism or scientific misconduct.
11

---

Published as a conference paper at ICLR 2026
REFERENCES
Mido Assran, Adrien Bardes, David Fan, Quentin Garrido, Russell Howes, Matthew Muckley, Am- mar Rizvi, Claire Roberts, Koustuv Sinha, Artem Zholus, et al. V-jepa 2: Self-supervised video models enable understanding, prediction and planning. arXiv preprint 250.::55060099855, 2025.
Gilles Baechler, Srinivas Sunkara, Maria Wang. Fedir Zubach, Hassan Mansoor, Vincent Etter, Vic- tor Cärbune, Jason Lin, Jindong Chen, and Abhanshu Sharma. Screenai: A vision-language model for ui and infographics understanding. arXiv preprint M.Xiv::2402.04615.. 2024.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-v1 technical report. arXiv preprint arXiv:2502. 13923, 2025.
Fengxiang Bie, Yibo Yang. Zhongzhu Zhou, Adam Ghanem, Minjia Zhang, Zhewei Yao, Xiaoxia Wu, Connor Holmes, Pareesa Golnari, David A Clifton, et al. Renaissance: A survey into ai text-to-image generation in the era of large model. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
Yingshan Chang, Mridu Narang, Hisami Suzuki, Guihong Cao, Jianfeng Gao, and Yonatan Bisk. Webqa: Multihop and multimodal qa. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16495-16504, 2022.
Qi Chen, Dileepa Pitawela, Chongyang Zhao, Gengze Zhou, Hsiang-Ting Chen, and Qi Wu. We- bvin: Vision-anndlanguagge navigation on websites. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 1165-1173, 2024.
Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiy- ong Wu. Seeclick: Harnessing gui grounding for advanced visual gui agents. arXiv preprint arXiv:2401.10935. 2024.
Jack Conklin. A taxonomy for learning, teaching, and assessing: A revision of bloom's 'ss taxonomy of educational objectives complete edition, 2005.
Xiang Deng, Yu Gu, Boyuan Zheng. Shijie Chen, Sam Stevens, Boshi Wang. Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 36:28091 281 14, 2023.
Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su. Navigating the digital world as humans do: Universal visual grounding for gui agents. arXiv preprint arXiv:2410.05243, 2024.
Izzeddin Gur, Ofir Nachum, Yingjie Miao, Mustafa Safdari, Austin Huang, Aakanksha Chowdhery, Sharan Narang, Noah Fiedel, and Aleksandra Faust. Understanding html with large language models. arXiv preprint 1..09:2200 2022.
Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. Webvoyager: Building an end-to-end web agent with large multimodal models. arXiv preprint arXiv:2401.13919 2024a.
Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Hongming Zhang, Tianqing Fang. Zhenzhong Lan, and Dong Yu. Openwebvoyager: Building multimodal web agents via iterative real-world exploration, feedback and optimization. arXiv preprint arXiv:2410. 19609, 2024b.
Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. Cogagent: visual language model for gui agents. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 14281-14290, 2024.
Yu-Chung Hsiao, Fedir Zubach, Gilles Baechler, Victor Carbune, Jason Lin, Maria Wang, Srinivas Sunkara, Yun Zhu, and Jindong Chen. Screenqa: Large-scale question-ana answer pairs over mobile app screenshots. arXiv preprint arXiv:2209.00 08/ 2022.
12

---

Published as a conference paper at ICLR 2026
Yi Huang, Jiancheng Huang, Yifan Liu, Mingfu Yan, Jiaxi Lv, Jianzhuang Liu, Wei Xiong, He Zhang, Liangliang Cao, and Shifeng Chen. Diffusion model-based image editing: A survey. IEEE Transactions Pattern Analysis and Machine Intelligence, 2025.
Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409. 12186, 2024.
Juyong Jiang, Fan Wang. Jiasi Shen, Sungju Kim, and Sunghun Kim. A survey on large language models for code generation. arXiv preprint arXiv:2406.005/3 2024.
Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po- Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. Visualwebarena: Evaluating multimodal agents realistic visual web tasks. arXiv preprint arXhv:2401.13649 2024.
Hanyu Lai, Xiao Liu, lat Long long, Shuntian Yao. Yuxuan Chen, Pengbo Shen, Hao Yu, Hanchen Zhang, Xiaohan Zhang, Yuxiao Dong, et al. Autowebglm: A large language model-based web navigating agent. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 5295-5306, 2024.
Guohao Li, Hasan Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Com- municative agents for" mind" exploration of large language model society. Advances in Neural Information Processing Systems, 06:51191-52000 2023.
Xiaoxi Li, Jiajie Jin, Guanting Dong, Hongjin Qian, Yutao Zhu, Yongkang Wu, Ji-Rong Wen, and Zhicheng Dou. Webthinker: Empowering large reasoning models with deep research capability. arXiv preprint arXiv:2504.21776 2025.
Evan Zheran Liu, Kelvin Guu, Panupong Pasupat, Tianlin Shi, and Percy Liang. Reinforcement learning web interfaces using workflow-guided exploration. arXiv preprint vrXiiv/802.0050 2018.
Junpeng Liu, Tianyue Ou, Yifan Song, Yuxiao Qu, Wai Lam, Chenyan Xiong, Wenhu Chen, Gra- ham Neubig, and Xiang Yue. Hamessing webpage uis for text-rich visual understanding. arXiv preprint ar arXiv:2410. 13824, 2024a.
Junpeng Liu, Yifan Song, Bill Yuchen Lin, Wai Lam, Graham Neubig, Yuanzhi Li, and Xiang Yue. Visualwebbench: How far have multimodal Ilms evolved in web page understanding and grounding? arXiv preprint 24i:22404.05955 2024b.
Yixin Liu, Kai Zhang, Yuan Li, Zhiling Yan, Chujie Gao, Ruoxi Chen, Zhengqing Yuan, Yue Huang, Hanchi Sun, Jianfeng Gao, et al. Sora: review on background, technology, limitations, and opportunities of large vision models. arXiv preprint 241. 2024c.
Liangbo Ning, Ziran Liang, Zhuohang Jiang, Haohao Qu, Yujuan Ding, Wenqi Fan, Xiao-yong Wei, Shanru Lin, Hui Liu, Philip S Yu, et al. A survey of webagents: Towards next- genetttenration ai agents for web automation with large foundation models. arXiv preprini 2:Xiv:2503.23350 2025.
CHRISTOPHER P Ormell. Bloom's taxonomy and the objectives of education. Educational Re- search, 17(1)3-18, 1974.
Yujia Qin, Yining Ye, Junjie Fang. Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, et al. Ui-tars: Pioneering automated gui interaction with native agents. arXiv preprint 12Xiv:2501.12326, 2025.
Ranjan Sapkota, Konstantinos I Roumeliotis, and Manoj Karkee. Ai agents vs. agentic ai: con- ceptual taxonomy, applications and challenge. arXiv preprint 200.25055.046. 2025.
Jianqiang Wan, Sibo Song, Wenwen Yu, Yuliang Liu, Wenqing Cheng, Fei Huang. Xiang Bai, Cong Yao, and Zhibo Yang. Omniparser: unified framework for text spotting key information extraction and table recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15641-15651 2024.
13

---

Published as a conference paper at ICLR 2026
Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. A survey on large language model based autonomous agents. Frontiers of Computer Science, 18(6):186345, 2024.
Hai-Ming Xu, Qi Chen, Lei Wang, and Lingqiao Liu. Attention-driven gui grounding: Leveraging pretrained multimodal large language models without fine-tuning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 8851-8859, 2025.
Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Pro- cessing Systems, 35:20744-20757, 2022.
Chaoyun Zhang, Shilin He, Jiaxu Qian, Bowen Li, Liqun Li, Si Qin, Yu Kang. Minghua Ma, Guyue Liu, Qingwei Lin, et al. Large language model-braince gui agents: A survey. arXiv preprint arXiv:2411.1827, 2024.
Pengsong Zhang. Xiang Hu, Guowei Huang, Yang Qi, Heng Zhang, Xiuxu Li, Jiaxing Song, Jiabin Luo, Yijiang Li, Shuo Yin, et al. aixiv: A next-generation open access ecosystem for scientific discovery generated by ai scientists. arXiv preprint arXivv2508..5/222 2025.
Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. Gpt-4v (ision) is generalist web agent, if grounded. arXiv preprint arXiv:2401.01614, 2024.
Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: A realistic web environment for build- ing autonomous agents. arXiv preprint arXiv:2307.1885. 2023.
14

---

Published as a conference paper at ICLR 2026
A COGNITIVE FRAMEWORK AND INTERACTION ENVIRONMENT
A.1 BLOOM'S s TAXONOMY AND COGNITIVE STAGES
A.1.1 BLOOM'S TAXONOMY
1. Factual Knowledge: The foundational layer, encompassing the basic, discrete elements of discipline that student must know, such essential terminology and specific, isolated details.
2. Conceptual Knowledge:
The synthesis of factual elements into coherent, organized structure. This level focuses the interrelationssiis between basic elements, including knowledge of classifications, principles, generalizations, theories, and models.
3. Procedural Knowledge:
The knowledge of how to perform a task or inquiry. This involves an understanding of specific skills, algorithms, techniques, and methods, representing shift from "knowing-what" to "knowing-how."
A.1.2 STAGES OF HUMAN KNOWLEDGE AND COGNITIVE DEVELOPMENT
Flighs
Now moser Are larke Lomait Any te Cheoe dight whfay
Figure 4: How people handle unfamiliar web pages. People learn factual, conceptual, and procedural knowledge to memorize, understand, and explore the web, ultimately completing specific tasks.
Cognitive science typically classifies human knowledge into three categories-factio conceptual, and procedural-which correspond to different stages of cognitive development: perceiving, under- standing, and executing. This taxonomy captures the natural trajectory of human learning: starting with the perception of concrete facts and data (factual knowledge), progressing toward abstract comprehension of concepts and relationships (conceptual knowledge), and ultimately acquiring the ability to carry out complex, goal-oriented behaviors through practiced routines and strategies (pro- cedural knowledge). An illustrative example is shown in Figure
A.2 WEB INTERACTION ACTION SPACE
Table summarizes the action space of Web-CogReasoner for web interaction tasks. The table cat- egorizes actions into two groups: (1) element-specific operations, such as clicking, typing, double- clicking, and scrolling individual elements; and (2) page-level control actions, including navigation commands, task restart, waiting, and final answer submission. This structured action space enables the agent to perform a diverse set of interactions, facilitating comprehensive exploration and manip- ulation of web pages in a controlled and systematic manner.
15

---

Published as a conference paper at ICLR 2026
Table 10: Action space of Web-CogReasoner for web interaction.
Instruction
Description
click [id]
Click element
type [id] [content]
Input specified content into an element Scroll an element or the page up/down
scroll [id or WINDOW] [op/down]]
dbclick [id]
Double-click an element
back
Navigate to the previous webpage
gorford
Navigate to the next webpage
stop [content]
Submit the final answer
Restart Wait
Restart the current task Wait for one second before proceeding
B WEB-COGDATASET: CONSTRUCTION AND DATA QUALITY
B.1 DATA SOURCING AND METADATA COLLECTION
To collect comprehensive metadata from web pages, we developed data collection tool based on Playwright. This tool performs deep traversal and interaction by systematically clicking on all elements within each page. We define each round of interaction (i.e., one click) as layer, and using this iterative approach, we collected Layer Layer data from 14 different websites (for complete website information, refer to Table
Table 11: Web elements's meta-data.
Data
Description
Element's CSS selectors
allcss
CSS selector sequence of preceding elements
outerhtml
element's outerhtml
location
element's 's boundingbox
role
element's role
name
element's name
For each web element, we capture its standalone screenshot, as well as screenshots taken before clicking (both with and without red bounding box), after hovering, and after clicking. See Figure S for an example. We also collect the following metadata: CSS, alICSS, outerHTML, and location. Additionally, we extract semantic information from each element based its outerHTML. If role attribute is explicitly defined, we use its value directly as the element's semantic role. Otherwise, we infer the role by mapping the tag the element's 's semantic name, we extract the value of the aria-label if present; otherwise, we get its textual content. See Table [ for detailed metadata of the web elements.
using the WAI-ARIA specification. Similarly, determine
After collecting both the visual and semantic metadata, we present the corresponding screenshots of each clickable element Qwen2.5-VL 72B. The model instructed
analyze the visual changes on the page after hovering over and clicking the target element;
identify and list any sub-elements that appear upon interaction (e.g., when dropdown menu is triggered by clicking);
infer and generalize the functional purpose of the element.
For the functional purpose prediction, the model is additionally required to provide a confidence from evaluation.
score. If this score remains below 0.5 after three retries, the prediction is excluded
16

---

Published as a conference paper at ICLR 2026
base_rect.pe
hover
Find deals for any season
treaifurr
Ieau
click png
masae aillyy Deke mvy
Figure 5: Example visual states of web element ("USD") we captured. Shown are: the element highlighted the full-page view rect.peec.. the hover state (hover.png). and the click state (click.png). These screenshots illustrate how the element's visual context evolves through user in- teractions.
Developer Tools 21 4%
Social Media
E-commerce
Finance
Developer Tools
Social & Media
28 6%
Amazon
Binance
Github
Zhihu
Finance
Hbay
Coinglass
Stack Overflow
Weibo
21 4%
E E-commerca ce 6%
12306
Eastmoney
Stack Exchange
Bilibili
Airhnb
Apple Music
Figure 6: Statistics of selected websites by category.
B.2 DATA COMPOSITION AND DOMAIN BALANCE
To avoid domain overfitting and ensure both interaction depth and broad generalization, Web- CogDataset employs strategic hybrid data composition:
1. Depth via Self-Collected Data: tion mining to capture intricate logic often missed by general crawls. As shown in Figure we strictly maintained category balance across E-commerce, Finance, Developer Tools, and Social Media within this subset prevent bias toward any single domain.
We selected 14 complex websites for nigh-depth interac-
2. Breadth via Open-Source Augmentation: To address the concern of limited domain di- versity (e.g., lack of News, Education, or Forums), we incorporated and enhanced large- scale open-source datasets, including MultiUI (Liu al. 2024a). Mind2Web (Deng al.) 2023). and OpenWeb Web Voyager (He et al.) 2024b). Notably, MultiUI is derived from Fine Web (Common Crawl), providing massive coverage of general- perrpoe webpages. This combination ensures our model generalizes to the "wild" web and is not overfitted to specific interaction styles like financial trading or shopping.
B.3 ANNOTATION RELIABILITY
We validated our automated annotations via double-bling human verification and cross-model con- sistency checks (e.g., using GPT-4o). As shown in Table 12, the rate is minimal.
17

---

Published as a conference paper at ICLR 2026
Table 12: Reliability Check of Web-CogDataset Annotations.
Annotation Task
Human Verification (Ace)
Crass-Model Consistency
Element Attribbuee
99.2%
98.5%
Page Change Pred
97.5%
96.8%
Sub-element Pred
96.8%
95.4%
Average
97.8% %
96.9 %
C DETAILED TASK SPECIFICATIONS AND STATISTICS
C.I. OVERALL DATASET STATISTICS
C.I.I WEB-COGDATASET
Table 13: Statistics Wet-SogDataset.
Knowledge
Task
Statistics
Subtotal
Total
Element Attribute Recognition
37K
Sub-element Prediction
1K
Factual Web Knowledge
Page Change Prediction
24K
81K
Next Page Prediction
18K
Source Element Prediction
IK
Element Understanding
40K
Conceptual Web Knowledge
WehPage Understanding
7K
62K
Caption & QA
15K
User's Intention Prediction Popup Close Single-Step Web Task
2K
IK
Procedural Web Knowledge
27K
17K
Noisy Multi-Step Web Task
7K
C.1.2 WEB-COGBENCH
Element Attribute Recognition: Given a screenshot with a highlighted interactive ele- ment, the model predicts its semantic role (e.g., button, link) and accessible name (e.g.. "Submit", "Search"), relying solely on visual cues.
Next Page Prediction:
The model predicts the subsequent page that results from interact- ing with a specific element on the current page. To enhance generalization, we design two types of tasks: multiple-choice questions and open-ended responses.
Source Element Prediction: Given two screenshots, the current page and the resulting target page, then the model identifies which of the visually marked elements on the current page leads the target, simulating visual cause-and-effect reasoning.
For a specific interactive element, the model generates an open- ended paragraph that comprehensively describes the element's Visible Traits (e.g., text, shape, styling), its On-page Location (e.g., header, sidebar, main content), and its likely User-facing Function (e.g., playing video, navigating new page), relying solely on visual context.
Element Understanding:
WebPage Understanding: Given full-page screenshot, the model generates a compre- hensive overview describing the webpage's Layout Organization (e.g.. header, event in- formation, seating chart, filter panel), Key Element Analysis (e.g., element attribute, de- scription, function, interaction, expected outcome), and a Summary of the WebPage. This enables thorough understanding of the webpage's structure and functionality.
18

---

Published as a conference paper at ICLR 2026
The model infers high-level user intent from sequence of webpage screenshots representing an interaction trajectory, requiring visual understanding and temporal reasoning. The task is built on the BolliMoodal-Mnnd2Wer (Deng al. 2023 dataset, mapping screenshot sequences to natural language instructions.
User's Intention Prediction:
The model identifies and dismisses popups (e.g., notification modals, login forms) on synthesized webpage screenshots. using dataset of 51 popup components from JS Design Desigrroverriidd overlaid on OpenWebVoyager (He et al. 2024b) webpages, with combinatorial augmentation of closing strategies for diverse training.
Popup Close:
This task derived from a multi-step trajectory exploration task and has been decomposed into single-step exploration subtasks. For each step. the model receives as input the corresponding accessibility tree (AxTree) and a screenshot of the current webpage. Based on these observations, the model performs reasoning to generate the appropriate action along with the target object, effectively simulating a realistic single-step web navigation scenario. By breaking down complex multi-step interactions into atomic actions, this setup allows for fine-graines evaluation of the agent's decision- making capabilities and supports systematic analysis of its performance in web exploration tasks.
Single Step Exploration:
Table 13 summarizes the overall statistics of Web-CogDataset. The dataset covers three layers of Factual, Conceptual, and Procedural- each corresponding to distinct families of web reasoning tasks. Factual Web Knowledge focuses recognizing attributes, predicting element re- lationships, and modeling page transitions, totaling 81K instances. Conceptual Web Knowledge emphasizes semantic understanding and cross-element comprehension, with 62K instances. Proce- dural Web Knowledge involves action-oriented reasoning tasks, such as predicting user intentions and executing goal-directed interactions, comprising 27K instances. Together, these task distribu- tions reflect the hierarchical design of Web-CogDataset and ensure balanced coverage from low- level perception to high-level reasoning.
poowledge--oo haw.ledeeaaauuaall
C.2 FACTUAL WEB KNOWLEDGE TASKS
Element Attribute Recognition We define the Element Attribute Recognition task assess model's 's capability to infer the interactive semantics of web elements exclusively from visual input. Given full-page screenshot with specific interactive element marked by red bounding box, the model is tasked with predicting two key attributes:
the semantic role of the highlighted element "button", "link", "checkhox").
the semantic name, which refers to the element's accessible textual description (e.g., "Sub- mit", "Search", "Next").
The ground truth for both attributes is derived from the element-level metadata collected as de- scribed in Section B.I This task simulates the human cognitive ability to interpret the function of web interface components through visual perception alone, without relying on HTML structure or programmatic representations.
Sub-elements Prediction We define the Element Sub-element Prediction task to evaluate model's ability to infer the hierarchical structure of web interfaces sub-elements that become visible upon interaction with a given parent element, using only visual information. In each task instance, the model is presented with full-page screenshot in which specific interactive element is highlighted by red bounding box. The model is instructed predict the set of sub-elements (e.g., menu items, dropdown options) that appear as a direct result of inter- acting with the highlighted element, such as clicking or hovering. The ground truth annotations for sub-elements are derived from the element-level metadata collected during the dynamic interaction process, as detailed in Section This task simulates the human cognitive process of understand- graphical interface interface-rrcogniiiee recognizing not only that component is
to identify the
ing interactive dependencies clickable, but also predicting its dynamic expansion behavior.
hhups:/jjs.esignn.
19

---

Published as a conference paper at ICLR 2026
Page Change Prediction We define the Page Change Prediction task to evaluate model's pability to infer the visual consequences of interacting with a specific web element, relying solely on visual input. In this task, the model is presented with full-page screenshot in which a tar- get interactive element is highlighted by a red bounding box. The model is required to prediet, in an open-ended format, the visual changes that are likely occur the page after the element is clicked. The ground truth for this task is obtained from the generated responses of Owen-VVL-2BB which were produced based on visual metadata, as detailed in SectionB.I This task designed to simulate the human cognitive ability to anticipate the dynamic behavior of web interfaces through perception alone without access to the underlying source code or prior knowledge of the page logic.
Next Page Prediction We define the Next Page Prediction task to evaluate model's ability to forecast navigation outcomes. Given full-page screenshot with a highlighted interactive element, the model must predict the subsequent page that would result from interacting with that element. To ensure generalization capability,
implement two evaluation formats: multiple-choiee selection (choosing from 4-5 possible next pages) and open-ended generation (describing the expected next page). Ground truth is derived from actual navigation sequences recorded during web interactions. This task assesses the agent's understanding of functional relationships between interface elements and destination pages.
Source Element Prediction We define the Source Element Prediction task to assess a model's ability to identify which element on a webpage leads specific target page, using only visual input. The model is given two screenshots: one showing the current webpage with 4-10 candidate elements marked by bounding boxes, and another showing the resulting target page. Based on visual cues alone, the model should determine which candidate element would trigger the transition to the target page when interacted with. This task simulates the human ability to reason about visual cause-and-effect relationships in web navigation, without relying on code prior knowledge of page logic.
C.3 CONCEPTUAL WEB KNOWLEDGE TASKS
Element Understanding This task requires the model to produce comprehensive, open-ended deseription of highlighted element's 's visual appearance, functional semantics, and placement on the webpage. Specifically, the output should cover: (1) Visual Traits (lext, shape, iconography); (2) Lo- cation (e.g., top right, footer); and (3) Function (e.g., navigates to user profile). This task simulates abstract comprehension from concrete element appearance.
WehPage Understanding In this task, the model must generate detailed and structured overview of the entire page. The response includes layout segmentation (e-g.. header, sidebar, content area), key modules (e.g., search panel, product gallery), and a summary of page purpose and interactivity. This facilitates understanding of page-wide structure and intent.
Caption & QA We define the Caption & QA task to evaluate model's capability to comprehend and reason over both image and non-image content embedded within webpages. This task comprises four subtasks:
Embedded Image Captioning: Given a full-page screenshot containing one or more em- bedded images, the model is required to generate a detailed and semantically meaningful caption for each image, describing its visual content and its contextual relevance within the surrounding webpage layout.
Embedded Image QA: Given question grounded in the content of an embedded image within a webpage screenshot, the model must produce an accurate, context-aware answer using only visual information. These questions may refer image content (e.g.. "What brand is shown in the ad?") or its function in the UI.
The model tasked with generating an open-ended description of the webpage's 's content, layout, and interactive purpose, treating the entire screenshot as input. The generated caption should reflect both structural composition and the inferred user intent of the webpage.
Webpage Captioning:
20

---

Published as a conference paper at ICLR 2026
Webpage QA:
Given fall-page screenshot and natural language question referring to any aspect of the page (e.g., title, layout, purpose, textual content), the model must generate grounded and precise answer based visual and spatial information.
All four subtasks are derived from the Multi-UI (Liu al. (024a) dataset, which provides rich an- notations for webpage visual elements and user-facing semantics. Together, these subtasks measure a model's ability to perform grounded visual-language understanding at both local (element-level) and global (page-level) scales.
C.4 PROCEDURAL WEB KNOWLEDGE TASKS
User's Intention Prediction Built the HittMModl--ind2WWW titaset-whice provides nat- ural language instructions, action trajectories, and aligned web page screenshots-wo introduce a novel multi-modal task: inferring the user's high-level intent from sequence of visual observations. Unlike traditional imitation learning or cnstruction-followee infer why trajectory occurred, rather than how to execute Solving this task demands both visual understanding and temporal reasoning. The details of this task are as follows:
tasks, our setting requires the model to
1. Task Definition: Given a sequence of web page screenshots P2......... Pr. representing user's interaction trajectory, the objective is predict the original user instruction y that guided the sequence. Each screenshot pt corresponds to the visual observation at step of successful task execution. Formally, the model learns a mapping: f .PI.PP.....PP.. Q where Q is the natural language instruction.
2. Dataset Construction: We construct our dataset by processing the original MultiModal- Mind2Web corpus. For each task, we extract only the visual observations-ice. the se- quence of web page screenshots corresponding each step in the execution trajectory. We then pair each screenshot sequence with the original natural language instruction as the supervision signal.
Popup Close We curated collection of 51 popup components from JS Design website, encom- passing a diverse range of visual styles and functional categories, such as notification modals, alert dialogs, and login forms. This diversity ensures comprehensive coverage of real-world popup cases, For background webpages, we utilized the Open Web Voyagee (He et al. (024b) dataset, which contains large number of authentic webpage screenshots with varied layouts and content, provid- ing rich foundation for synthesizing realistic popup- injeeeneectted webpages. To construct the training data for this task, we employed the following procedure:
1. Synthesizing Webpage Screenshots with Popups: We randomly overlaid popup images onto background webpage screenshots to simulate webpages containing popups. During synthesis, we introduced variability by randomly adjusting the popup's size and position and modifying the brightness and sharpness of the background images, thereby enhancing visual diversity and realism.
2. Generating Popup AX Tree: Each popup image was processed using Owen-VV-2.55-2 to generate ARIA-compliant AX Tree. To simulate diverse structural configurations, we randomly modified the index values of popup AX Tree elements and inserted the popup AX Tree into different locations within the original webpage's 's AX Tree, resulting in combined AX Tree that reflects realistic variations in webpage structure.
3. Generating Popup Closing Strategies: We then instructed Qwen- Qwen-VL-2.55 VL-2.5--2B to identify all n possible methods for closing the popup, based on the popup image and its correspond- ing AX Tree. Recognizing that, in practical settings, any correct method is sufficient, we applied combinatorial augmentation the methods. Specifically, we enumerated all non- empty subsets of the strategies, yielding a total of 2n 1 distinct answer combinations. This expansion significantly broadens the training distribution and increases the model's exposure to diverse correct solutions.
4. Constructing the Training Dataset: Using the synthesized webpage screenshots and the enriched AX Tree, we constructed a dataset for training models on popup dismissal. Each data point comprises:
21

---

Published as a conference paper at ICLR 2026
Input:
webpage screenshot with embedded popup and the corresponding AX
Tree;
Output: valid methods for closing the popup.
Single-Step Web Task We define the Single-Step Web Task to evaluate a model's ability to ground high-level user intentions in visual webpage elements. Each task instance includes full-page screenshot from a real-world webpage, a concise natural language instruction (e.g., "Search for a product", "Log into the system"), and several candidate elements marked by red bounding boxes.
The model must identify which element, if clicked, would successfully fulfill the given task. This setup simulates perceptual grounding of user intent-matching natural language goals to actionable UI targets based solely on visual cues.
All samples are directly sourced from the Multi-UI (Liu et al. 2024a) dataset, which provides rich, annotated webpage screenshots paired with task descriptions and labeled ground-truth targets. No trajectory-level annotation or external instruction rewriting involved. This task offers a reliable benchmark for evaluating atomic web interaction capabilities in a static, visually grounded setting.
Noisy Multi Step Web Task To further enhance the original Open Web Voyager (He et al. 2024b) dataset, we incorporate Knowledge-driven Chain-of-Thought (Co') Reasoning improve the model's stepwise understanding and execution, details see Section 4.2 In addition, to simulate realistic interruptions during multi-step web interactions, we propose the Noisy Multi-Step Web Task by augmenting interaction trajectories from Open Web Voyagr,ger. Specifically, for each sample in our Popup Close dataset, a popup window is injected at a specific step (e.g.. step t) of an existing task trajectory.
This modification introduces prerequisite interaction: the agent must first detect and dismiss the popup before resuming progress toward the original task goal. By explicitly modeling such interrup- tive UI elements, this task formulation captures a more realistic web interaction paradigm which user flows are frequently obstructed. It also provides challenging benchmark for evaluating agents' robustness to UI-level noise and their capacity for error recovery.
D TRAINING METHODOLOOGIES AND EVALUATION PROTOCOL.
D.1 TRAINING IMPLEMENTATION AND STRATEGIES
We employ a multi-phase Imitation Learning strategy to train our model on Web-CogDataset, uti- lizing Qwen2.5-VL. VL-7B (Bai et al. 2025) as the base model. Each phase is aligned with distinct layer of the Web-CogKnowledgs Framework: (1) the first knowledge content learning focuses on acquiring Factual Knowledge and Conceptual Knowledge, enabling the model to interpret web con- tent and semantics; (2) the second cognitive process emphasizes Procedural Knowledge, training the model to plan and execute multi-step web interactions. To accommodate the increased complexity of the final phase, which involves multi-image inputs and extended reasoning, we configure training with maximum sequence length of 8K and a batch size of I with gradient accumulation of 16 steps. All experiments are conducted on a cluster of x NVIDIA A800 80GB GPUs.
We could feed these SFT datasets into models with different training strategies. In this section, We investigate how they influence the model's performance.
1. Curriculum Learning Strategy: We fine-tune the model following a curriculum learning paradigm.
2. Mixed Multi-task Learning: We directly mix different tasks and apply SFT.
Through comparative analysis, we find that: Under the same task scenario, models trained via cur- riculum learning conduct multiple rounds of exploration continuously and do not exploration prematurely. In contrast, when retrieving task-related data, models trained via mixed training termi- nate the search within limited number of attempts if they fail to find results.
22

---

Published as a conference paper at ICLR 2026
As shown in Figure 7 when faced with the user's instruction "Open the most helpful 5 star reviews of Alpine Ridge", models trained via mixed training deviate from the domain of task specific in- formation retrieval and autonomously generate instruction to "switch to forums for information".
Based on these observations, we conjecture: Curriculum learning trains models incrementally from simple to complex tasks, which enables them develop fundamental understanding of the prob- lem structure. They continue to carry out multiple rounds of exploration to ensure comprehensive comprehension and accurate processing, thus avoiding the premature termination of exploration. In contrast, mixed training models may be trained on multiple tasks simultaneously. The poten- tial interference among different tasks can make models easily influenced by irrelevant tasks when processing a specific one, thereby undermining their ability to focus on task specific information retrieval and processing.
Casel: Open the helpful star reviows of Alpine Ridge.
Histary
Hlateey
nonking
wnon con way duald vetuinghe Bmars
Bugiivd Dec colion leats)
Figure 7: Comparison between mixed and curriculum strategies.
D.2 LVM-BASED EVALUATION PROTOCOL
For open-ended generation tasks in Web-CogBench (i.e., Element Understanding, WebPage Under- standing, and User Intent Prediction), we utilize high-capability LVM (GPT-40) automated evaluator. The evaluation process involves a strict comparison between the Candidate Model's Answer and the verified Ground Truth based on image provided in the dataset.
To ensure fine-grained assessment, we decompose the evaluation into specific cognitive dimensions rather than relying on single holistic score:
Element Understanding:
Assessed on Appeurance (visual fidelity), Position (structural context), and Function (interaction purpose).
Assessed on Structure & Layout, Key Element Analysis, and Summary Coherence.
WebPage Understanding:
Assessed on Evidence Alignment (visual cue detection), Intent Accuracy, and Reasoning Quality.
User Intent Prediction:
We employ a rigorous 1-5 Likert Scale and normalized to 0-100 for scoring. A score of 5 denotes that the candidate "fully and accurately captures all relevant information present in the Ground Truth," while lower scores reflect varying degrees of omissions or inaccuracies. The evaluator out- puts structured ISON object containing both integer scores and text justifications for each dimen- sion, ensuring the traceability of the results.
For instance, the Flement Understanding task is assessed on:
23

---

Published as a conference paper at ICLR 2026
You are a meticulous and impartial Al ovaluator for a web UI understanding benehmark. Your task is to assess the quality candidate model's answer by comparing strictly against ground truth and provided image reference. Your evaluation must be based exclusively on the information provided in the "Ground Truth Answer" and "Image". specific aspects: Appearance, Position, Function.
System Instruction:
answer [Ground Truth Auswer]
fground )
[Candidate Model's Answer]
[modellannwerr
Evaluation Criteria & Scoring:
Completely
missing.
Score 2: Mosily incorrect, with minor element of truth.
Partially currect,
significuni details
ground
Score 4: Mostly correct, with only minor inaccuracies or emissions compared to the ground truth.
Fully accurately captures relevant information present the ground
Your response MUST be single, valid ISON object, adhering the following stricture. Do addd any text hefore after the ISON object.
appearance, score": jinteger score), popearance.jstfficatioo Your brief justifiication... referencing the ground truth,""
"position score": jinteger
"pasitionjuusificaaiii justification" "; Your brief justification... referencing the ground iruthg",
"function_scome" sccore" sinteger_scorej,.
"function. dumntion.ustificationn
"overall justificationn: YYur brief brief justiication... referencing the ground iruthg". ;A holistic integer score to 5g. "overall, "ustilljutiffcation" ";A final summary of the model's performance;""
D.3 EVALUATOR RELIABILITY ANALYSIS
To mitigate potential biases from single evaluator, we employed multiple distinct LVMs (in- cluding GPT-4o, Claude Sonnet 4, and Gemini 2.5 Pro) to conduct a rigorous inter-rater reliability analysis. We calculate the "Within-1-Poin Agreement", defined as the percentage of instances where scores assigned by different LVM judges differ by no more than 1 point.
As shown in Table 14 the high agreement rates across different models confirm that our evaluation criteria are robust and model-agnostic, Furthermore, the strong correlation with Human Proxy Anal- ysis suggests that our Ground-Tree Truth Anchored protocol effectively aligns automated judgment with human evaluation standards.
Table 14: Inter-Rater Reliability Analysis of LVM Judge.
Task
Within-I-Point Agreement
Human Proxy Analysis
Element Understanding
98.7%
96.
WebPage Understanding
97.0%
95.4%
User Intent Prediction
96.0%
94.4%
E EXTENDED QUALITATIVE ANALYSIS
To provide deeper insights into Web-CogReasoner's capabilities, we present two-part qualitative analysis. First, we examine the evolution of cognitive abilities across training stages to validate our curriculum. Second, we present a comparative case study on a complex real-world task to demonstrate how model overcomes knowledge blind spots that trap baseline models.
E. EVOLUTION OF COGNITIVE ABILITIES ACROSS STAGES
Beyond the quantitative improvements shown in our ablation study, a qualitative analysis of the agent's behavior at each stage offers deeper insights into how curriculum shapes its cognitive abilities. We examine the agents' performance on a representative task: "Find and add laptop under $1000 to the shopping cart e-commerce website."
24

---

Published as a conference paper at ICLR 2026
Base Model (Qwen2.5-VL-77) Without any specialized training, the base model struggles to formulate a coherent plan. Its reasoning is often generic and untethered from the specific UI. It might correctly identify a "search bar" but fails to execute meaningful action, hallucinates actions that are not possible. For instance, its thought process might be: "I should search for a laptop," but its action is an ungrounded 'click "Categories" because it lacks the procedural knowledge to connect intent to multi-step sequence of actions.
Stage 1 Agent (+ Factual Knowledge) After training Factual Knowledge, the agent's 's per- ceptual abilities are significantly enhanced. It can now accurately identify and label key elements with their correct attributes. Its thought process becomes grounded in the facts of the page: "I see a search bar [ID: 25] with the name 'Search products'. I see button [ID: 28] with the name 'Search'. :" However, it still struggles with planning. It understands "what" on the page but not "why" or "how" to use it. It might correctly type "laptop" into the search bar but then get stuck, not understanding that the next logical step is to click the search button to submit the query.
Stage 2 Agent (+ Conceptual Knowledge) With the addition of Conceptual Knowledge, the agent
begins to understand the relationships between elements and their purpose. Its reasoning graduates from simple identification to semantic interpretation. The thought process now reflects this under- standing: "The search bar [ID: 25 is for imputting queries. The search button [ID: 28] is function- ally linked to it and will trigger the search. This group of elements forms a 'search component""" This allows it to reliably complete the search and navigate to the result page. However, on the result page, it may still struggle with complex procedural logic, such as applying price filter.
Full Model (Web-Cog + Procedural Knowledge) The final agent, equipped with Pro- cedural Knowledge, demonstrates goal-oriented planning and execution. It seamlessly translates the high-level task into a concrete action sequence. Its thought process is now a strategic plan: "Goal: Add laptop under $1000 to cart. Step 1: Type 'laptop' into search bar [ID: 25J. Step 2: Click search button [ID: 28). Step 3: On the results page, locate the 'Price Range' filter. Step 4: Input '1000' into the 'max price' field [ID: 57]. Step 5: Identify suitable product from the filtered list and click its 'Add to Carl' button IID: 83)." This demonstrates a complete cognitive loop from perception and understanding successful action, validating the necessity of the final procedural training stage.
E.2 COMPARATIVE CASE STUDY ON REAL-WORLD TASKS
To validate the necessity of foundational knowledge in handling complex real-world scenarios, we compare the Base Model with Web-CogReasoner specific Amazon task.
Task: "Find a gaming desktop with Windows 11 Home and 1TB disk."
The Knowledge Blind Spot Lacking explicit knowledge of page layout and element functions, the Base Model literally "sees" the pixels but "misses" the affordance.
Base Model Failure:
The model sees the search results but fails to recognize the sidebar filters as the mechanism refine the query.
Observation:
Error: It misinterprets the page state, assuming a re-search is necessary. It enters a infinite loop of repeatedly clicking the search button.
Action: click [1470] (Search Button) + Stuck.
Knovledge-Driven Grounding Leveraging learned Factual and Conceptual knowledge, our agent explicitly identifies the functional role of UI components.
Success:
The agent identifies the sidebar filter section. It conceptually maps the user's "ITB" requirement to the specific filter element, predicting that clicking it will nar- row the results without leaving the page.
Reasoning:
Action: cl click [95] (Fi (Filter Link "1 TB") Success.
This comparison highlights that success in complex tasks is not just about planning (Procedural), but requires strictly accurate recognition of element functions (Factual/Concepuua)) to ground those plans.
25


Reviews:
[{"review_id": "7iHoKdd1N0", "reviewer": "Reviewer_ri9L", "summary": "The paper proposes Web-CogReasoner, a web agent trained with a two stage, knowledge first paradigm inspired by Bloom’s taxonomy: first learn Factual, Conceptual, and Procedural web knowledge, then exercise cognitive processes for reasoning and action. The authors build Web-CogDataset from 14 real sites with 12 fine-grained tasks, and introduce Web-CogBench to evaluate “Memorizing, Understanding, Exploring” capabilities aligned to those knowledge types. A  Web-CogReasoner is further trained with Qwen2.5-VL-7B as base model and  achieves strong performance on benchmarks including Web-CogBench and VisualWebBench.", "questions": "Listed above.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The evaluation of Web-CogBench is aligned to the training data construction framework.  Web-CogBench measures “Memorizing, Understanding, Exploring,” directly mirroring the training objectives and enabling targeted diagnosis of capabilities.\n\nThe combination of screenshots with the accessibility tree leverages both visual cues and structured semantics of web pages.", "weaknesses": "Authors have placed much emphasis on the three aspects of “factual, conceptual, and procedural learning,” claiming that “higher-order reasoning is built on solid perceptual and conceptual foundations.” It would be great to see supporting experiments. \n\nIt would be helpful to provide more details around the selection of “memorizing, understanding, and exploring” abilities as the tasks for Web-CogBench. Some questions that may be good to answer are: how well do these aspects cover the actual tasks performed by web agents. Are these necessary conditions for a strong web agent.\n\nIt would be good to know the reason behind choosing webvoyager and online-mind2web as the only two end to end web agent benchmarks, as they both rely on live websites and environments, which can bring noise to the experiment. In particular, it would be good to know the reason behind choosing online multimodal mind2web over multimodal mind2web. And it would be great to provide more details around experiment setup, e.g. is human or LLM judge used as evaluation. If Web-CogReasoner is set to take the role of a deep research style agent, then live environment with benchmarks such as GAIA could be a good choice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PR72KPJASe", "reviewer": "Reviewer_HEcV", "summary": "This paper proposes a Web-CogKnowledge framework inspired by Bloom's taxonomy, where web navigation knowledge is categorized into three levels and a two-phase training methodology is followed in order to enhance the cognitive capabilities of web agents. Based on this framework, the authors construct Web-CogDataset and use this dataset to create Web-CogBench. Furthermore, the authors train a Web-CogReasoner model and conduct evaluations against several popular models on multiple benchmarks.\n\nOverall, this is an interesting paper attempting to frame web agent learning processes from a cognitive perspective following Bloom's taxonomy. However, all core ingredients are well-explored by related work under different names, resulting in insufficient technical novelty.", "questions": "- Regarding the POMDP formulation – how is the Markovian property maintained when previous action history is included in the 'trajectory history' section? Doesn't this violate the memoryless assumption?\n- In Web-CogDataset, how do you define the single-step and multiple-step web tasks? What's the source?\n- Can more details about the LVM Judge be disclosed?\n- How is accuracy measured for open-ended responses in next page prediction?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Strong motivation and interesting idea to frame the problem using Bloom's taxonomy, attempting to model the learning process as three progressive stages (Memorizing, Understanding, Exploring) that map to different knowledge types.\n\n- The dataset is well-structured following the proposed framework, covering 12 fine-grained tasks across factual, conceptual, and procedural knowledge, which could be a valuable contribution to the community for training models and conducting evaluations.", "weaknesses": "### Major\n\n- The three proposed stages are essentially different names for well-established concepts like visual grounding, VQA, and action prediction that are widely used in current web agents, just reframed under cognitive terminology.\n\n- To strength the argument of the proposed framework, the ablation study should demonstrate performance with and without each individual stage, rather than progressively adding stages one by one, to fully investigate the independent impact of each component in the proposed framework.\n\n- Evaluations should be more comprehensive, missing some key benchmarks: static grounding tasks such as ScreenSpot series and dynamic environments such as WebArena and VisualWebArena that better reflect real-world web interaction scenarios.\n\n- Web-CogBench remains static in nature and doesn't capture the dynamic, interactive aspects of real web environments where agents must adapt to changing page states and handle unexpected scenarios\n\n### Minor\n\nLine #343: 'via upervised fine-tuning' should be 'via supervised fine-tuning'", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zJUbWriZmr", "reviewer": "Reviewer_PZyc", "summary": "This paper proposes Web-CogReasoner, a knowledge-induced cognitive reasoning framework for web agents. The key idea is to structure agent reasoning with a knowledge-driven chain-of-thought (K-CoT) aligned to three layers—factual, conceptual, and procedural—and to train a Qwen2.5-VL-7B–based agent via staged supervision on a new dataset (Web-CogDataset) collected from real websites. A companion benchmark (Web-CogBench) evaluates three capability dimensions—memory, understanding, and exploration—that map to the knowledge layers. Experiments report strong performance on Web-CogBench and competitive results on established web-agent benchmarks (e.g., VisualWebBench, WebVoyager, Mind2Web / Online Mind2Web), with ablations showing complementary gains from all three knowledge layers. These design choices are well-motivated by trends in web-agent evaluation and environment standardization.", "questions": "**Reliance on pseudo-labels and their reliability**: Some annotations are generated by Qwen-VL-72B; the paper should report the error rates, whether they performed human double-blind verification, or whether cross-model annotation consistency was checked (e.g., compare labels from Claude / Gemini).\n\n**Judge bias / evaluator correlation**: The use of an LVM Judge for scoring may be correlated with the model training biases or data domain. Do authors provide inter-judge agreement or bootstrap confidence intervals, especially on the Understanding / Exploring tasks?\n\n**Limited demonstration of active exploration / online feedback**: While procedural knowledge and exploration are core aims, the paper heavily leans toward structured CoT and imitation learning. More experiments incorporating online decision-making (e.g. RL fine-tuning, exploration during inference) would strengthen claims about interactive capability.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "**Clear, systematic framework**: The use of Bloom’s taxonomy to structure the agent’s learning and reasoning is conceptually coherent. While knowledge-layered modeling is not entirely novel, applying it concretely to web agents with CoT alignment is a valuable contribution.\n\n**Well-structured data and evaluation system**: The pairing of Web-CogDataset and Web-CogBench, mapped by knowledge layers and ability dimensions, allows end-to-end evaluation from knowledge acquisition to interactive behavior.\n\n**Reproducibility awareness and engineering detail**: The paper reports training setups, hardware, sequence lengths, and strategy choices, and includes responsible research statements. This transparency helps reproducibility and fosters trust.", "weaknesses": "**Website sampling bias**: The chosen 14 sites may disproportionately lean toward e-commerce or financial domains. This could bias the model’s strengths toward those styles; broader site diversity (e.g., social media, education, news, forums) would strengthen generalization.\n\n**Lack of strong baseline comparisons with GPT-level models**: Without direct comparisons to large generalist models (e.g. GPT-4, GPT-5), it is hard to assess how far this approach is from “state-of-the-art general reasoning agents.”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GCLQOuDKNO", "reviewer": "Reviewer_YsZ8", "summary": "The paper Web-CogReasoner: Towards Knowledge-Induced Cognitive Reasoning for Web Agents proposes a novel cognitive learning framework for web agents, inspired by Bloom’s taxonomy of human learning. The authors argue that effective web agents must acquire three types of knowledge—Factual, Conceptual, and Procedural—to perform reasoning and exploration tasks on the web. They introduce three key components: Web-CogKnowledge Framework, Web-CogDataset, and Web-CogBench, enabling a systematic curriculum that teaches agents to memorize, understand, and explore. The resulting model, Web-CogReasoner, integrates knowledge-driven chain-of-thought (CoT) reasoning, achieving 84.4% accuracy on Web-CogBench, outperforming Gemini 2.5 Pro, Claude Sonnet 4, and prior open-source agents on reasoning-intensive tasks", "questions": "1. How would Web-CogReasoner perform if trained with reinforcement learning or self-play instead of pure imitation, to strengthen procedural adaptability?\n\n2. Could the authors include more diverse model families and scales (e.g., 7B–70B, open- and closed-source) to validate generality across architectures?\n\n3. Have the authors explored integrating multimodal cognition—such as combining visual-textual reasoning with real-time interaction on dynamic or partially observable webpages?\n\n4. To what extent is the Bloom’s-inspired hierarchy empirically necessary—would a non-hierarchical or single-stage variant achieve comparable performance?\n\n5. Can the framework be extended toward lifelong or self-directed learning, enabling agents to autonomously acquire new procedural knowledge from unstructured online experiences?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "This work presents a conceptually rich and methodologically rigorous framework that advances the frontier of web-agent cognition. By grounding web reasoning in educational psychology (Bloom’s taxonomy), the authors provide a human-analogous paradigm for hierarchical learning—progressing from factual perception to procedural reasoning. The proposed dataset and benchmark cover 14 real-world websites with fine-grained multimodal tasks, yielding both scalability and interpretability. The experiments are comprehensive and ablation-supported, demonstrating clear contributions from each knowledge layer. The structured chain-of-thought (CoT) design substantially improves cross-task generalization and reasoning interpretability, and the authors provide transparent reporting and reproducibility details. Overall, the work represents a significant step toward cognitively inspired, knowledge-grounded web agents.", "weaknesses": "Despite its novelty, the study has several limitations. The model diversity is narrow—experiments primarily rely on Qwen 2.5 VL-7B, without testing multiple scales or architectures (e.g., Llama, Claude Opus, GPT-5), which constrains external validity. The framework’s dependence on supervised imitation learning raises concerns about long-horizon exploration, and its performance gap in cross-web generalization (≈10%) suggests limited adaptability beyond curated domains. Additionally, the theoretical connection between Bloom’s taxonomy and CoT reasoning is conceptually interesting but lacks formal grounding or cognitive validation. The framework also does not address multimodal robustness under dynamic webpages or visual perturbations, and scalability to open-world, continuously evolving web environments remains uncertain.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)