modify robots.txt and add notification not to use thunder