robotxt (cargo): The implementation of the Robots.txt (or URL exclusion) protocol with the support of crawl-delay, sitemap and universal match extensions. 
Keywords: parser|builder|robots|crawler|txt|rfc-9309|robots-exclusion-protocol|robots-exclusion-standard