changed robots.txt to allow for crawlers