Back to using sites to determine current url for robots.txt.