crwlr/robots-txt (packagist): Robots Exclusion Standard/Protocol Parser for Web Crawling/Scraping
Keywords: parser|scraper|spider|crawler|bot|robots.txt|robots|protocol|standard|scraping|bots|exclusion|crwlr|hacktoberfest|robots-exclusion-protocol|robots-exclusion-standard|robots-txt|robots-txt-parser|web-crawling|web-scraping