whitelist search engines that drive traffic, block other bots that consumes bandwidth for their own purpose