parse robots.txt
Header names are case insensitive
Move header checks to web_site