# An MLLM-Assisted Web Crawler Approach for Web Application Fuzzing
## 2 Background and Motivation
### Crawler-Based Web Application Fuzzing  
Web crawlers serve as fundamental components in modern web application security testing, particularly during fuzzing operations. Their significance is demonstrated through four essential capabilities: (1) automated test target discovery, (2) systematic test case generation, (3) dynamic scope determination, and (4) comprehensive vulnerability analysis support. These automated systems methodically traverse web applications, enumerating all accessible resources--including static pages, dynamic content, interactive forms, API endpoints, and auxiliary elements--establishing a comprehensive testing interface for fuzzing operations. Through sophisticated path exploration algorithms, crawlers generate context-aware test cases that cover diverse testing scenarios, ensuring thorough coverage of input vectors and user interaction patterns. During operation, crawlers maintain detailed execution logs and test results, providing security analysts with critical data for identifying potential vulnerabilities and abnormal behaviors. These comprehensive logs serve as invaluable resources for analyzing fuzzing results and diagnosing complex security issues.  
In the cybersecurity domain, web crawlers serve as critical tools for comprehensive state discovery and security assessment. These automated crawling systems initiate from designated seed websites and employ iterative browsing techniques to explore various pages and resources within Web applications. Crawler operations primarily rely on two distinct strategies: navigation algorithms and page similarity algorithms [12]. Within navigation algorithms, random algorithms and Breadth-First Search (BFS) represent common approaches, with random BFS emerging as an enhanced hybrid algorithm that improves upon traditional BFS through the incorporation of randomization strategies. This methodology combines systematic exploration with adaptive selection strategies, enabling crawlersto dynamically adapt to variations in website architectures and content loading mechanisms. Empirical evidence demonstrates that random BFS algorithms, in particular, exhibit superior performance in terms of average coverage and efficiency. For content deduplication purposes, page similarity algorithms primarily utilize URL matching and DOM-based analysis techniques. URL matching has demonstrated marked advantages in large-scale data deduplication tasks through its superior processing speed, algorithmic simplicity, and scalability, consistently outperforming DOM-based methods across the majority of performance metrics.  
While web crawlers demonstrate significant potential in vulnerability scanning applications, they face several fundamental technical challenges. These challenges primarily arise from the inherent complexity of modern Web applications, which constrains the crawlers' capability to achieve comprehensive path and state coverage, particularly when encountering sophisticated authentication mechanisms and access control policies. Furthermore, although current page similarity algorithms effectively address data redundancy issues, they face substantial challenges in optimizing the trade-off between deduplication efficiency and test coverage during intensive large-scale operations. To overcome these limitations, current research efforts focus on enhancing crawler systems through the strategic integration of artificial intelligence techniques, with the objective of achieving higher precision and efficiency in vulnerability detection.