Management of robots crawl rules in header