Package: se-bot-checker
Search Engine Bot CheckerThis is a simple python library that verifies the validity of a search engine crawler based on it's IP and user agent.It is designed to assist SEO's and DevOps validategooglebotand other search engine bots.Installationpip install se-bot-checkerUsageUsing SE Bot Checker to validate a search engine crawler is simple. There are two basic steps.Instantiate the bot class.Call the bot class with IP and user agent arguments.fromse_bot_checker.botsimportGoogleBotgooglebot=GoogleBot()test_one=googlebot('66.249.66.1','Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)')test_two=googlebot('127.0.0.1','Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)')print(test_one)print(test_two)Output:(True, 'googlebot')
(False, 'unknown')Prebuilt BotsThere are several bot definitions that are already created, have been tested and will be maintained. The prebuilt
crawlers are the most common search engine crawlers.Crawler validation methodsBotUser AgentIPDNSBaiduSpiderXX*X**BingBotXX*XDuckDuckBotXXGoogleBotXX*XYandexBotXX*X* IP validation is only used on consecutive checks run using the same bot checker instance. This means that in the
following example there will be only one DNS network request since the IP intest_twohas already been validated whentest_onewas run.** BaiduSpider only supports reverse DNS validation not reverse and forward. Although it on first glance it appears
BaiduSpider should support reverse/forward DNS validation I have never had forward success for BaiduSpider.fromse_bot_checker.botsimportGoogleBotgooglebot=GoogleBot()test_one=googlebot('66.249.66.1','Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)')print(test_one)# (True, 'googlebot')test_two=googlebot('66.249.66.1','Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)')print(test_two)# (True, 'googlebot')BaiduSpiderBaiduSpider validation only uses reverse DNS lookup not reverse and forward.Name:baiduspiderDomains:.baidu.com,.baidu.jpUser Agents:baiduspiderUse RegEx:FalseBingBotName:bingbotDomains:.search.msn.comUser Agents:bingbot,msnbot,bingpreviewUse RegEx:TrueDuckDuckBotDuckDuckBot only uses IP validation from the list of valid IPs.Name:duckduckbotIPs:See list belowUser Agents:duckduckbot,duckduckgoUse RegEx:True20.191.45.212
40.88.21.235
40.76.173.151
40.76.163.7
20.185.79.47
52.142.26.175
20.185.79.15
52.142.24.149
40.76.162.208
40.76.163.23
40.76.162.191
40.76.162.247
54.208.102.37
107.21.1.8

Updated: January 31, 2022GoogleBotName:googlebotDomains:.googlebot.com,.google.comUser Agents:googlebotUse RegEx:FalseYandexBotName:bingbotDomains:.search.msn.comUser Agents:bingbot,msnbot,bingpreviewUse RegEx:TrueCreating Your Own Bot DefinitionSE Bot Checker was designed to be extensible. The core of SE Bot Checker is theBotclass. To create your own
bot you can simply extendBot.Here is custom bot that will only validate Googlebot mobile.fromse_bot_checker.botsimportBotclassMobileGoogleBot(Bot):"""Mobile googlebot checker"""name='googlebot-mobile'domains=['.googlebot.com','.google.com']user_agent='android.*googlebot'That is all there is to it. However, we could simplify this a little by extending theGoogleBotclass.fromse_bot_checker.botsimportGoogleBotclassMobileGoogleBot(GoogleBot):"""Mobile googlebot checker"""name='googlebot-mobile'user_agent='android.*googlebot'Both the desktop and mobile versions of Googlebot use the same domains for the reverse/forward DNS validation. This
means we can simply extendGoogleBot. This is the recommended approach when possible.BotAPIThis class is the core of SE Bot Checker. It handles the validation process. New bot definitions should subclass this
class.A single bot class can be instantiated once and called many times. The allows base settings to be configured and
multiple IP and user agent pairs to be validated simply.Bot.name:strThis is the name the bot will return if it validates toTrue.Bot.ips:iterableA list of known valid IPs.Bot.domains:iterableA list of known valid domains. This is used to validate the results of the reverse
DNS lookup. An exact match or a super domain of the DNS lookup results is considered a positive match.Bot.user_agent:strA substring or RegEx pattern to use to validate the request user agent. For the best
performance and compatibility request user agent string are changed to lowercase prior to matching. theuser_agentstring should be lower case. If you need to validate upper or mixed case user agents you can override theBot.valid_user_agent()method.Bot.use_regex:boolWhether the user agent validation should use substring or regex matching. Ifuser_agentis just a string and not a RegEx pattern this should beFalse. It slightly faster. Defaults toFalse.Contributors@danielmorellCopyright © 2020Daniel Morell