Adding note about robots.txt ignoring.