wwweb (npm): An autonomous webcrawler for indexing robots.txt files.
Keywords: software-agent|autonomous|bot|webcrawler|robots