robot-directives (npm): Parse robot directives within HTML meta and/or HTTP headers.
Keywords: crawlers|header|html|http|meta|metadata|nofollow|noindex|robots|robots.txt|seo|spiders|attributes|headers|nodejs