GitHub repository: xwde/robotxt
Primary language: Rust
Description: The implementation of the Robots.txt (or URL exclusion) protocol with the support of crawl-delay, sitemap and universal match extensions.