robots-parser (npm): A specification compliant robots.txt parser with wildcard (*) matching support.
Keywords: robots.txt|parser|user-agent|scraper|spider|bot|robots-exclusion-standard|javascript|nodejs|robots-parser|robots-txt