GitHub repository: Dev-29/sitemap-crawler
Primary language: Go
Created: 2024-04-14T17:30:34Z
Description: A Go-based sitemap crawler for extracting SEO data from websites. This tool automates the discovery and parsing of sitemaps to retrieve URLs and gather key SEO metrics such as page titles, H1 tags, and meta descriptions. It's designed to support concurrency, adhere to robots.txt protocols.