robotsparser (pypi): Parse robots.txt files and find indexed urls
Keywords: robots|txt|sitemap|crawler|data|mining