updated configuration to speed up crawls