robots-txt-parser (npm): A lightweight robots.txt parser for Node.js with support for wildcards, caching and promises.
Keywords: robots|txt|robots.txt|parser|crawler|spider|bot|robotstxt|scraper|javascript|node-js|robots-exclusion-standard|robots-parser|robots-txt|robots-txt-parser