Add check for robots.txt
Extract header_dict function