added some settings to prevent our spiders getting banned