update the spider and don't respect robots.txt