Added functionality to allow for site specific robots.txt