stricter rate limit
update robots.txt to prevent spidering

this is a precaution, as I'm not sure if the spam links are being
scraped by spiders