robotstxt (cran): A 'robots.txt' Parser and 'Webbot'/'Spider'/'Crawler' Permissions Checker
Keywords: crawler|peer-reviewed|r|r-package|robotstxt|rstats|scraper|spider|webscraping