GitHub repository: KrishivGubba/Concurrent-WebCrawler
Primary language: Java
Created: 2024-07-10T08:06:06Z
Description: Java Web Crawler Project: This repository implements a multi-threaded web crawler in Java, designed to fetch and parse web pages concurrently. It includes classes for managing URL queues, tracking visited URLs, parsing HTML content, and handling data storage, rate limiting, and robots.txt compliance.