# User-agent strings
Inktomi
Slurp
WebCrawler
GoogleBot
Googlebot
ZyBorg
Openbot
Scooter
ia_archiver
UrlDispatcher
Ask Jeeves
larbin
MarkWatch
www.walhello.com
Netprospector
Crawler
Ultraseek
AlkalineBOT
libwww-perl
bumblebee@relevare.com
Inet32 Ctrl
3D_SEARCH
LinkLint
Teleport
vspider
BaiDuSpider
TITAN/
GetRight/
WebTrends Link Analyzer
MyApp/
SystemSearch-robot
contype
lwp-trivial/
lwp-request/
LexiBot
linkbot
cosmos/
webcollage
byteserver/
NetAnts/
RaBot
NEC-MeshExplorer
Crawl_Application
Aport
nabot_
Verity-URL-Gateway
WebSpider
viasarchivinginformation.html
searchengine
autoemailspider
Xenu
WebStripper
Robot
Web Downloader
WebCopier
BizWorks Retriever
Pompos/
Gulliver/
ASPSeek/
FirstGov.gov Search
psbot/
Sqworm/
MOSES
WebWasher
CacheBlaster/
FlashGet
wwwxref/
VoilaBot/
Gaisbot/
htdig/
grub-client
Grubclient-
SQ Webscanner
SlySearch/
Mercator-
gsa-crawler
DOF-Verify/
WebCapture
EasyDL/
Rotondo/
InternetLinkAgent/
Gigabot/
DigOut4U
Cyberdog/
ASPseek/
http://www.almaden.ibm.com/cs/crawler
eCatch/
indylabs_marius
b2w/
AnswerChase PROve
Mass Downloader/
Teradex Mapper
Lycos_Spider
RPT-HTTPClient/
WebBOT
Szukacz/
GaisBot/
targetblaster.com/
Infoseek Sidewinder/
InfoSeek Sidewinder/
Infoseek SideWinder/
DiaGem/
Steeler/
Fluffy the spider
searchhippo.com
webbandit/
webfetch/
ah-ha.com crawler
Lytranslate/
Vagabondo/
moget/
WebZIP/
Robozilla/
Oracle Ultra Search
w3m/
EmailLeach
OrangeBot
iCab/
metacarta
TurnitinBot/
DISCoFinder
InfoLink/
crawler_for_infomine
minibot
NetResearchServer/
EmailWolf
eidetica.com/spider
polybot
HomePageSearch
spider
WebFetch
AnswerBot
UdmSearch
NG/
3DE_SEARCH2
Mozilla/3.0 (compatible)
Mozilla/3.01 (compatible;)
DoCoMo/
URLBlaze
Ad Muncher
Net Vampire/
EyeNetIE
www.galaxy.com/info/crawler.html
Net Vampire/
DoCoMo/
nabot
Pita
Knowledge Engine
Webclipping.com
SSM Agent
INGRID/
Mister PiX
IAArchiver-
DISCo Pump
NPBot-
Python-urllib/
Zeus
Lachesis
lachesis
HLoader
Caddbot/
Zao/
WWW-Mechanize/
Nessus
VoilaBot
Wget/
WebReaper
LARBIN-EXPERIMENTAL efp@gmx.net
Goldfire Server
Mozilla/2.0 (compatible; NEWT ActiveX; Win32)
T-H-U-N-D-E-R-S-T-O-N-E
thatrobotsite.com
NPBot
WebSauger
Mozilla/3.0 (compatible; Indy Library)
WebGather
MFHttpScan
NutchOrg/
SiteFinder/
search.usgs.gov
TestBot
Space Bison/
Sleipnir
QuepasaCreep
MSNBOT/
pavuk/
VSE/
SearchSpider.com/
SpiderKU/
Dattatec.com-Sitios-Top
Microsoft Internet Explorer/4.40.426
WebVac
YahooSeeker/
infomine.ucr.edu
SiteSweeper
PageFetcher/
msnbot/
GetHTMLContents
Cowbot-
freshlinks.exe
Cowdog Bot
Art-Online.com
AnswerBus
MultiText/
Dattatec.com
NaverBot-
Panopy Bot
ez-robot
Scich/
Dumbot
Web Magnet
LinkWalker
TranSGeniKBot
LinkScan/
Links SQL
Google/
antibot-
Wotbox/
SpiderEngine
X-clawler
X-crawler
TygoBot
TulipChain
NetResearchServer
LinkSweeper/
Z-Add Link Checker
web-agent/
NaverBot_dloader/
mini-robot/
www.searchnear.com
Botswana
nuSearch Spider
slurp
CrawlConvera
geometabot/
kuloko-bot/
Webdup/
JoeDog/
Dillo/
Ocelli/
GPU p2p crawler
TheUsefulbot_
Unitek UniEngine/
COAST scan engine/
BlackMask.Net Search Engine
Offline Explorer/
EAH/
Spider.TerraNautic.net
Scrubby/
Check&Get
IECheck
GoForIt.com
sohu-search
NuSearch Spider
Search-Channel
NutchCVS/
LinkAlarm/
CSE HTML Validator
oBot
WISEbot/
boitho.com-dc/
EMPAS_ROBOT
Jetbot/
iVia Site Checker
findlinks/
USS-Cosmix/
mod_accessibility
ht://check/
LookBot
CheckLinks/
Html Link Validator
osis-project.jp
Seekbot/
Iltrovatore-Setaccio/
Website Downloader
BDFetch
CreativeCommons/
Nutch
SOFT411 Directory
Yandex/
ClariaBot/
MSNPTC/
heritrix/
HiSoftware AccVerify
Diamond/
OmniFind
Download Ninja
Artera
contype
FAST-WebCrawler/
Yahoo-VerticalCrawler-FormerWebCrawler/
falcon/
Shareaza
ZipppBot/
WorQmada/
Web Link Validator
CFNetwork/
CyberSpyder
FAST Data Search Document Retriever/
ImageBot
ABACHOBot
Microsoft Office Protocol Discovery
Microsoft Data Access Internet Publishing Provider Cache Manager
EmailSiphon
searchmarks
f-bot test pilot
Antro.Net
www.sygol.com
HSFT - Link Scanner
w3search
Buscaplus Robi/
adminshop.com
statcrawler
Clushbot/
Blinkx/DFS-Fetch
Header_Test_Client
fast-search-engine
WebXM
WebPix
DiamondBot
Express WebPictures
www.superbargainspace.com
archive.org_bot/
omnifind
statbot
Combine/
SpiderMan
searchmarking/
SiteXpert
ContentSmartz
gazz/
FindAnISP.com_ISP_Finder
CCGCrawl/
TravelBot/
NSDL_Search_Bot
pipeLiner/
Webinator-WBI/
KummHttp/
crawler.kpricorn.org
versus.integis.ch
search.updated.com
websearchbench
Cafi/
LinkChecker/
Microsoft_Site_Analyst/
webcrawl.net
HTTrack
Buibui
Stumbler
athenusbot
HiSoftware AccMonitor Server
MetaGloss
SURF
TREX
Validator/
HooWWWer/
www.thebananatree.org/
Spinne/
Favorites Sweeper
MJ12bot/
maxomobot/
KnowItAll
VisWeb
WebIndex/
BruinBot
ShowLinks/
Acme.Spider
Nogate
CD-Preload
websphinx.test
Missigua Locator
ApacheBench
RSSbot/
Yahoo-Newscrawler/
P3P Client
googlebot
WebAuto/
SiteSnagger
Plucker/
NetShift=
websitemirror
Eventware/
Akamai-SiteSnapshot/
zagrebin
theusefulbot
inetbot/
iSiloX/
hcat/
gnome-vfs/
didaxusbot
WebTrafficExpress/
Talkro Web-Shot/
TPSystem
SuperBot/
SpaceBison/
SiteSucker/
SEB Spider
Poodle predictor
PhpDig/
PageRank Monitor
NASA Search
MonkeyCrawl/
Mag-Net
Lament/
LMQueueBot/
K.S.Bot
Hooblybot-Image/
HTMLParser/
GoodJelly/
DiGi-RSSBot
CydralSpider/
Amfibibot/
AONDE-Spider/
moduna.com/
BecomeBot/
YottaCars_Bot/
DOY/
Forest Conservation Spider
Patwebbot
tankvit@e-mail.ru
Eco-Portal Spider
EcoEarth Portal
sherlock/
Knowledge.com/
smartwit.com
W3C-WebCon/
Digger/
USyd-NLP-Spider
ichiro/
Link Checker/
accoona
EduGovSearch/
WinHTTP Example/
FDM 1.x
unchaos_crawler_
TygoProwler
grub crawler
IRLbot/
GigabotSiteSearch/
FreshDownload/
WinHttp.WinHttpRequest.
ClimateArk Spider
BoaConstrictor/
CyberNavi_WebGet/
updated/
wish-la
Custo
InelaBot/
oegp v.
Water Conserve Spider
labourunions411/
SpeedySpider
FindWeb
OmiExplorer_Bot/
btbot/
Govbot/
tScholarsBot
OmniExplorer_Bot/
www.unchaos.com
NWSpider
www.nameprotect.com
cfetch/
AtlocalBot/
abot/
SCrawlTest/
Forests.org Spider
aipbot/
MojeekBot/
DataFountains/DMOZ Downloader
Speedy Spider
McBot/
BigCliqueBot/
eStyleSearch
ProjectWF-java-test-crawler
Rational SiteCheck
wgao@genieknows.com
Download Master
Kitenga-crawler-bot
BigCliqueBOT/
Ocean Conserve Spider
snap.com beta crawler
IlTrovatore-Setaccio/
Blaiz-Bee/
Metager2
GOFORITBOT
genevabot
versus crawler
Filangy/
Twiceler
Eco-Portal http://www.environmentalsustainability.info/
Yahoo-MMAudVid/
nicebot
Squid-Prefetch
parasite
cookieNET
PageBitesHyperBot/
Dir_Snatch.exe
CipinetBot
WebZIP
Arachmo
ccubee/
IEAutoDiscovery
Bilbo/
WebZIP
CE-Preload
endeca
HSFT - LVU Scanner
rssImagesBot/
Twisted PageGetter
Space Fung/
BrowserEmulator/
Microsoft Scheduled Cache Content Download Service
locust
PoCoHTTP
Web Site Downloader
Seeker.lookseek.com
Norbert the Spider
searchbot
grapeFX/
CSHttpClient/
AIBOT/
Miva
iVia/
WebIndexer/
Microsoft URL Control
AF Knowledge Now Verity Spider
MemacBot
ExactSearch
UofTDB_experiment
www.Syntryx.com
Charlotte/
NewMedhunt/
Metaspinner/
LeechGet
yacy.net
MSRBOT/
SygolBot
Zyte/
axfeedsbot/
testbot
OnetSzukaj/
LinksManager.com_bot
Helix/
MyEngines-US-Bot
maxamine.com--robot
SBIder/
Newsgroupreporter
IpselonBot/
byindia/
Xerka WebBot
Xerka MetaBot
Der groe BilderSauger
Tecomi Bot
fgcrawler
TutorGigBot/
yoono/
LocalcomBot/
dtSearchSpider
WebMiner/
wbdbot
via translate.google.com
WebDownloader for X
wume_crawler/
Eco Earth Spider
Checkbot/
Thumbnail.CZ robot
National Park Service Dan Buan
KATATUDO-Spider
xirq/
Tarantula/
its-learning crawler
Cuasarbot
Ipselonbot/
Deepindex
DataSpearSpiderBot/
libiViaCore/
generate_infomine_category_classifiers
RufusBot
sohu agent
Everest-Vulcan Inc./
Yahoo-Blogs/
focused_crawler
SocSciBot
SiteArchive
WebMiner
wacbot
ObjectsSearch/
Pockey-GetHTML/
OutfoxBot/
SearchBlox
PHPCrawl
STEGMANN-Bot
Plumtree 6.0;
AChulkov.NET page walker
SearchIt-Bot/
crawler@
LetsCrawl.com/
COAST WebMaster Pro/
voyager/
AVSearch-
InspireBot
Eco Earth Portal
genieBot
Theophrastus/
MyFamilyBot/
fr_crawler
Myra
Wavefire/
Forest Conservation Portal,
1Noonbot
ActiveTouristBot
DataSpider/
MSRBOT
Kyluka krawl
Water Conserve Portal
Cerberian Drtrs
silk/
BOI_crawl_00
Vortex/
ZoomSpider
KSbot/
BuyHawaiiBot
Arikus_Spider
ImagesHereImagesThereImagesEverywhere/
GruBot
OpenIntelligenceData/
Evaal/
cvaulev
c r a w l 3 r
Forschungsportal/
LBot
Girafabot
combine/
COMBINE/
www.dlese.org
virus-detector
web crawler
Link Checker
Scumbot/
SearchSpider.com
GulperBot
Wysigot
AccMonitor Compliance Server
DTAAgent
Y!J-BSC/
NetSongBot/
full_breadth_crawler
DataFountains/
AISIID/
Vermut
WebCorp/
Poirot
searchmee_v
Syntryx ANT Scout Chassis Pheromone; Mozilla/4.0 compatible crawler
PerMan Surfer
rsssupport@repia.com
Climate Change Portal http://www.climateark.org/
zimeno/
webcrawler
RAMPyBot
YahooSeeker-Testing/
vermut +http://vermut.aol.com
pucl/
personal ultimate crawler
Snoopy
__TBJ_WEB_CRAWLER__
TERAGRAM_CRAWLER
www.octora.com
exactseek.com
VIP/
Exabot-Images/
BuildCMS crawler
mercuryboard_user_agent_sql_injection.nasl
Linkman
wwwster/
AdamM Bot,
webbot
bzBot/
worldshop/
search.msn.com/msnbot.htm
NewsGatherer/
EcoEarth.Info Environment Portal
bot/
FurlBot/
ScollSpider;
Favcollector/
WIRE/
NLese
Feedfetcher-Google;
FeedBurner/
Jakarta Commons-HttpClient/
IlTrovatore/
INFOMINE/
Mammoth/
Searchmee! Spider
Crawl/
BecomeJPBot/
Sokitomi crawl;
http://www.sokitomi.com/crawl.html
Megite
Geomaxenginebot
Skywalker
VIPr/
WEBCRAWLER@VUNET.ORG
KeepNI web site monitor
MFcrawler
1on1searchBot/
Yeti
SuperPagesBot/
Search Publisher
seeqpod-vertical-crawler
NewsTroveBot
MQbot
metaquerier.cs.uiuc.edu
Big Brother
Froola Bot
webGobbler/
zedzo.validate/
zedzo.digest/
FedContractorBot/
EARTHCOM.info/
!Susie
Harvest/
BaiduSpider
virus_detector
Oracle Secure Enterprise Search
FlashCapture
Alpha Search Agent
yoogliFetchAgent
yarienavoir.net/
topicblogs/
robotek
page_verifier
online link validator
augurfind
Selflinkchecker
scSpider/
KakcleBot
JetBrains Omea Pro
HyperEstraier/
Gigabot
12soso/
pixfinder/
JoyScapeBot/
BySpider
DXSeeker/
psycheclone
bottybot
Exabot-Test/
Exabot-XXX/
Nusearch Spider
MaSagool/
WeRelateBot/
Fetch API Request
TargetYourNews.com bot
robots/
MAINSEEK_BOT
Lydia Entity Spider
Link Validator
BIGLOTRON
Bookmark Buddy bookmark checker
UniversalSearch/
imds_monitor/
BilgiBetaBot/
VMBot/
crawl@digigetx.com
perform_crawl
HSlide/
Vital Search'n Urchin
AESpider/
NaverBot/
METASpider
SumitBot
SquidClamAV_Redirector
AstroFind/
QFKBot
Website Quester
DigExt; DTS Agent
Weddings.info Bot/
ozelot/
Webinator-search2.fasthealth.com/
PsBot
qualidade/
WEPA/
Blog Conversation Project;
factbot
ODP entries t_st;
BilgiBot/
YahooFeedSeeker/
statedept-crawler
YahooFeedSeeker Testing/
Exploder/
miniRank/
Spider
wastrix/
TridentSpider/
arianna.libero.it
WebImages
Touche
StackRambler/
PicoSearch/
NetMechanic
vlad/
JobSpider_BA/
InsumaScout/
Fopper
Chameleon/
Climate Ark http://www.climateark.org/
info seeker/
vlsearch (http://vlib.org/admin/robot)
MnoGoSearch/
Pagebull http://www.pagebull.com/
http://pressemitteilung.ws/
ConnectSearch
SurveyBot/
Factbot
iVia Page Fetcher
LinkCheck Scanner/
IU_CSCI_B659_class_crawler/
SurfControl
kinjabot (http://www.kinja.com)
Rondello/
FDSE robot
SrevBot
holmes/
IlseBot/
gsa-accuracyEval
Qweery_robot.txt_CheckBot/
NLESE USEPA
BLT/
Earth Science Educator  robot
crawl@globrix.com
Nerima-crawl-
TerrawizBot/
DataparkSearch/
Gordon-College-Google-Mini
TravelLazerBot/
PrivacyFinder/
cs-crawler +http://citeseer.ist.psu.edu
del.icio.us-thumbnails/
LT Scotland Checklink/
nextthing.org/
nys-crawler
Y!J-PSC/
Blogslive
JemmaTheTourist
Web-Sniffer/
(www.cotse.net; Anon Proxy)
ShopWiki/
pythonic-crawler (suzuki@tkl.iis.u-tokyo.ac.jp)
crawler43.ejupiter.com
QweeryBot/
Net::Trackback/
BYINDIA/
FU-NBI/FU-NBI-
IIITBot (pvvpr@iiit.net)
Y!J-SRD/
Hatena Antenna/
RedCarpet/
GurujiBot/
RSS_READER (mctwist@mail.dr-k.info)
WebaltBot/
GigaBot/
AASP/
Whirlpool Web Engine
OmniWeb
http://www.mozilla.org/docs/en/bot.html; master@mozilla.com
AboutUsBot/
TRAAZI/
masidani_bot_
LamerExterminator/
Lsearch/sondeur
Yoono; http://www.yoono.com/
core-project/
zibber-v
(www.zibb.com/crawler/)
Pansophica/
iSearch/
ImageVisu/
Yoriwa/
kulturarw3 +http://www.kb.se/kw3/
bladder fusion
1-More Scanner
OutfoxMelonBot/
GeoVisu/
BoardReader-Image-Fetcher
cataguru/
FunnelBack; http://cyan.funnelback.com/
LM Harvester
Metaeuro Web Search
Treezy/
Mozilla/2.0 (compatible; MSIE 4.0; Windows 98)
DataCha0s/
wish-project (http://wish.slis.tsukuba.ac.jp/)
AlexfDownload
My_Little_SearchEngine_Project/
Vacobot; (+http://vaco.ws/bot.html)
http://www.t6labs.com//
SumeetBot
my-heritrix-crawler(
yellowJacket/
ChemieDE-NodeBot/
Koninklijke Bibliotheek web archive (heritrix +http://www.kb.nl)
BlogMyWay.Net/BlogMyWay-0.8.1 (admin@blogmyway.org)
MYCOMPANYBOT
RoboPal (http://www.goldcave.com/)
Heritrix/
LapozzBot/
domaincrawler/
ScientificCommons.org/
OpenISearch/
HarvestMan
Yahoo-Test/
Ocean Conserve http://www.oceanconserve.org/
gsa (Enterprise; GIX-
Pogodak.co.yu/
Trovator
heritrix bot
ExaBotTest/
Little Grabber at Skanktale.com
USAF AFKN K2SPIDER
kbeta1 +http://www.kotoha.co.jp
Exabot Test/
SiteOrbiter
Interseek/
SnapPreviewBot
Kyluka crawl; http://www.kyluka.com/crawl.html; crawl@kyluka.com
here will be link to crawler site
testing of bot;
PiyushBot
ROBOT
Tailrank; http://tailrank.com/robot
PythonWikipediaBot/
AntiSantyWorm
StarDownloader/
fembot (myd@cs.stanford.edu)
VisBot/
penthesila/
ucb-nutch/
FeedChecker/
Btsearch/
libcrawl/
grbot
Semager/
imo-google-robot-intelink
AlexaWebSearchPlatform; +http://websearch.alexa.com
http://www.changedetection.com/bot.html
TeezirBot/
Canon-WebRecordPro/
pulseBot (pulse Web Miner)
FDM 2.x
Faviconizer
crawler/
Hyperix/
wenbin/search
LeapTag/
WebSpear/
Dit/
wwwrobot
iim_405/
VadixBot
SkreemRBot +http://skreemr.com
Bigado.com/
Search-Engine-Studio
OsO; http://oso.octopodus.com/abot.html
+http://www.convera.com)
TapuzBot/
GalaxyBot/
WikiaBot
nestReader/
LolongBot/
mozilla (nlmoccssearchadmin@mail.nlm.nih.gov)
Dwaar crawler (dwaarbot@dwaar.com)
semanticdiscovery/
maxamine.com-robot
LeapTag (
IIITBOT/
Sphider2
MSMOBOT
http://www.artiesoft.com/lexxbot.php
Francis/
RcStartBot
Intelix/
Snappy/
imagefortress +http://www.worldbank.org)
woriobot (+http://worio.com)
opidig_1.0 (dfuhry@cs.kent.edu)
BuscadorClarin/
voyager-partner-deep/
Blackbird/
CazoodleBot/
Obvius external linkcheck/
Sphere Scout&v
GoogleReport Search Engine - http://www.googlereport.org
Attributor.comBot
Dwaarbot (dwaarbot@dwaar.com)
NatchCVS/
(Natch; http://lucene.apache.org/natch/bot.html; natch-agent@lucene.apache.org)
Obvius external linkcheck/
TsWebBot/
Sphere Scout
PIENO robot
robot; http://www.xrss.eu/robot;
Webscan
+http://otc.dyndns.org/webscan/
Bot; http://www.activetourist.com
Canon-WebRecord/
HD nutch agent/
Distilled-Reputation-Monitor/
slow-crawler +http://casr.ou.edu
WebBot/
Camcrawler (+http://www.camdiscover.com/crawler.html)
mowserbot; http://www.mowser.com/bot
Anonymous/3G bot
optidiscover/
Giant/
(Openmaru bot; robot@openmaru.com)
ASAHA Search Engine Turkey
BlogPulseLive (support@blogpulse.com)
Charlotte
DiBot
Elblindo the Blind Bot
MT-Soft (http://www.mt-soft.com.ar)
Mediapartners-Google
Mozilla/5.0 (FunnelBack)
Jim +http://www.hanzoarchives.com)
SearchnowBot_v1; +http://www.searchnow.com)
SummizeBot +http://www.summize.com)
Wazzup1.0.
pogodak.ba/
PWeBot/
PageDown/
froGgle/
kinjabot
medrabbit/
optidiscover/
semisearch/
stero (http://www.stero.pl;
News_Search_App/
Giant/
(Openmaru bot; robot@openmaru.com)
Compatible;Viking/
BlogRefsBot/
SummizeFeedReader +http://www.summize.com
Wazzup1.0.4800; http://32.fb.354a.static.theplanet.com/Wazzup)
dejan/
hul-wax +http://hul.harvard.edu/ois/projects/webarchive/)
sslbot +http://www.networking4all.com)
rtgibot; http://rtgi.fr/)
owsBot/
(owsBot; www.oneworldstreet.com; owsBot)
Sapienti/Indexer
KeywenBot/
Chilkat/
woriobot (+http://www.worio.com/)
REBI-Shoveler/
LiteFinder/
gsa (Enterprise;
wastrix/
CRAWLER-ALTSE.VUNET.ORG-Lynx
you-dir/
voyager-hc/
DjangoTraineeBot/
BSearchR&D/
webLyzard/
testBOT/
suchclip
(Kalooga; http://www.kalooga.com; info@kalooga.com)
fetch_ici/
Wikio (http://www.wiko.fr)
UCLA%20Google%20Serch%20Appliance%20%232%20%28contact%3A%20
UCLA%20Google%20Serch%20Appliance%20%231%20%28contact%3A%20
sslbot +http://www.networking4all.com)
hul-wax +http://hul.harvard.edu/ois/projects/webarchive/)
http://32.fb.354a.static.theplanet.com/Wazzup)
Najdi.si/
EARTHCOM/
Giant/
(Openmaru bot; robot@openmaru.com)
superbot.com; +http://www.super.info)
(Exabot-Thumbnails)
Zotag Search
egothor/
SafariBookmarkChecker/
sportcrew-Bot
(Grub.org crawler; http://www.grub.org/; bot@grub.org)
search x-bot
The Dyslexalizer @ http://spunc.dsturgeon.net
taxinomiabot
DAUMOA-video; +http://ws.daum.net/aboutkr.html
Website Explorer/
iHWebChecker
WEP Search
Google Keyword Tool; +http://adwords.google.com/select/KeywordToolExternal)
Mozilla/3.0 [en] (AWV2.72f)
Proximic crawler; +http://www.proximic.com/en/about-us/contact-us.html)
OpiDig
DAUMOA-video; +http://ws.daum.net/aboutkr.html)
quest.durato/
(Suchmaschine der durato Ltd.; http://quest.durato.de; quest@durato.de)
kalooga/
(Kalooga; http://www.kalooga.com; info@kalooga.com)
Zotag Search
Atomic_Email_Hunter/
besserscheitern-crawl
WatzBot
Verifactrola/
Toplistbot
TopServer PHP
Webwasher/
TSM Translation-Search-Machine (www.ttn.ch)
REBI-shoveler/
(REBI's great worker; http://rebi.co.kr; noreply@rebi.co.kr)
DAUMOA-web; +http://ws.daum.net/aboutkr.html)
Yahoo-Kids/
mailto:vertical-crawl-support@yahoo-inc.com)
link_checker/
Bot Apoena
http://www.katatudo.com.br/ajuda/
AmPmPPC.com (http://www.ampmppc.com/)
Topicalizer/www.topicalizer.com)
FormulaFinderBot/
AmPmPPC.com (+http://www.ampmppc.com/)
Runnk RSS aggregator : http://www.runnk.com/
G10-Bot/
autowebdir 1.1 (www.autowebdir.com)
D1GArabicEngine/
crawlmaster@d1g.com)
travel-search
GoSeebot; +http://www.gosee.com/bot.html)
Earth Platform Indexer
wikiwix-bot-
EnaBot/
ninetowns
woriobot +http://worio.com)
Mozilla/4.0 (compatible; MSIE 4.0; Windows NT; ....../1.0 )
Getleft
GetLeft
ShablastBot
Quintura-Crw/
TinEye/
(http://tineye.com/crawler.html)
(XML Sitemaps Generator
iearthworm/
iearthworm@yahoo.com.cn
C:\Documents and Settings\Joe\Desktop\HARVEST EMAILS\SEGMENTS\
GeonaBot/
BloobyBot
iearthworm/
YesupBot/
; +http://www.yesup.net/bot.html)
CCBot/
(+http://www.commoncrawl.org/bot.html)
HMSEbot
crawler-upgrade-config
JoBo/
Daumoa/
proximic; +http://www.proximic.com)
ScoutJet; +http://www.scoutjet.com/)
InfoUSABot/
Attributor/Dejan-
(Test crawler; http://www.attributor.com; info at attributor com)
exooba/exooba crawler (exooba; exooba)
zermelo; +http://www.powerset.com) [email:paul@page-store.com,crawl@powerset.com]
BitvoUserAgent (+http://www.bitvo.com) Bitvo/
zermelo; +http://www.powerset.com)
culsearch/culs/
(crawl@citeulike.org)
Hostcrawler
Sunrise XP/
AnotherBot
wisponbot(http://www.wispon.com,mailto:wispon@theory.snu.ac.kr)
woriobot support [at] worio [dot] com +http://worio.com)
GrubNG
swish-e http://swish-e.org/
search.KumKie.com
SeeqBot +http://www.seeqpod.com
YebolBot
zermelo/
+http://www.powerset.com/about/zermelo)
hijbul-heritrix-crawler (+http://mobide.korea.ac.kr/)
Knight/
(Zook Knight; http://knight.zook.in/; knight@zook.in)
Sphider
BpBot/
(- -; http://blitzpost.com; search@blitzpost.com)
FeedHub MetaDataFetcher/
attributor/
+http://www.attributor.com)
Kyluka crawl; http://www.kyluka.com/static/crawl.html; crawl@kyluka.com)
TeamSoft WinInet Component
DotSpotsBot/
(crawler; support at dotspots.com)
msnbot-products
FeedFetcher(www.radian6.com/crawler)
ScooperBot www.customscoop.com
Yahoo Pipes
3GSE bot (Internet Research Institute UK, http://iri-uk.com)
Kyluka crawl; crawl@kyluka.com; http://www.kyluka.com/static/crawl.html)
192.comAgent
+http://www.evri.com/evrinid)
Google Keyword Tool; +https://adwords.google.com/select/KeywordToolExternal)
CamelStampede/
eSyndiCat Bot
crawl.UserAgent
ShadowWebAnalyzer (http://www.safety-lab.com/)
Vishal For CLIA/clia-alpha-testing (Crawling for CLIA project ; www.cfilt.iitb.ac.in; vishalv@cse.iitb.ac.in)
CollapsarWEB qihoobot@qihoo.net)
Yanga WorldSearch Bot
Runnk online rss reader : http://www.runnk.com/ : RSS favorites : RSS ranking : RSS aggregator  hybridwse@runnk.com
BlitzBOT@tricus.net
; ODI3 Navigator)
Axonize-bot
DotBot/
Yanga WorldSearch Bot
Vishal For CLIA/
(Crawling for CLIA project ; www.cfilt.iitb.ac.in; vishalv@cse.iitb.ac.in)
OOZBOT/
(--; http://www.setooz.com/oozbot.html; agentname at setooz dot_com)
Runnk online rss reader
wauuu engine/Wauuu
(wauuu engine; http://www.wauuu.com; wauuu@wauuu.com)
NESSUS::SOAP
healia/healia (the personalized health search engine.; http://www.healia.com)
Climate Ark - http://www.climateark.org/)
^Byte (http://CaretByte.com)
A1 Sitemap Generator/
(+http://www.micro-sys.dk/products/sitemap-generator/) miggibot
JadynAve - http://www.jadynave.com/robot
JadynAveBot; +http://www.jadynave.com/robot
Drupal (+http://drupal.org/)
(vBSEO; http://www.vbseo.com)
crawly@commandcom.com
FatBot
http://www.thefind.com/crawler)
CoolCheck
iCopyright Conductor
Firebat
(http://lms.virtual-presence.org)
Google Bot 2 Beta
ornl_crawler_1
Mozilla crawl/
(compatible; frt/
BlogScope/
+http://www.blogscope.net/; U of Toronto)
snookit/Snookit (domains@snookit.com)
NetID.com Bot
IOI/
(ISC Open Index crawler; http://index.isc.org/; bot@index.isc.org)
betaBot
xqrobot
RIIGHTBOT/RIIGHT-
(riight.com; http://www.riight.com/riightbot; riightbot@riight.com)
Y!J-BRI/
crawler ( http://help.yahoo.co.jp/help/jp/search/indexing/
Grub/
(IOI crawler; http://index.isc.org/; crawl@index.isc.org)
Labhoo+(+http://www.labhoo.com/)

# Host names
offense.sses.net
.crawl.yahoo.net
.netvigator.com
crawler.bloglines.com
.verity.com
user.connect.gpo.gov
mail.encharter.org
sandbox-d.gsfc.nasa.gov
sandbox-qa1.gsfc.nasa.gov
gcmdstage.gsfc.nasa.gov
www.mvspy.com
doi-esn-gw.customer.alter.net
.keymachine.de
p6043-ipbfp305yamaguchi.yamaguchi.ocn.ne.jp
122.54.176.147.pldt.net
64-28-180-114-rev.cernel.net
200.87.117.125
123.226.146.43
210.180.187.248
12.192.176.244
219.134.139.39
124.244.194.27
124244194027.ctinets.com
85.19.205.0
162.140.67.10
.ask.com
.maxamine.net
.abhsia.telus.net
simple5.dragonara.net
c30099.upc-c.chello.nl
ABTS-North-Dynamic-031.46.161.122.airtelbroadband.in
116.30.36.170