# User-agent strings
Inktomi
Slurp
WebCrawler
GoogleBot
Googlebot
ZyBorg
Openbot
Scooter
ia_archiver
UrlDispatcher
Ask Jeeves
larbin
MarkWatch
www.walhello.com
Netprospector
Crawler
Ultraseek
AlkalineBOT
libwww-perl
bumblebee@relevare.com
Inet32 Ctrl
3D_SEARCH
LinkLint
Teleport
vspider
BaiDuSpider
TITAN/
GetRight/
WebTrends Link Analyzer
MyApp/
SystemSearch-robot
contype
lwp-trivial/
LexiBot
linkbot
cosmos/
webcollage
byteserver/
NetAnts/
RaBot
NEC-MeshExplorer
Crawl_Application
Aport
nabot_
Verity-URL-Gateway
WebSpider
viasarchivinginformation.html
searchengine
autoemailspider
Xenu
WebStripper
Robot
Web Downloader
WebCopier
BizWorks Retriever
Pompos/
Gulliver/
ASPSeek/
FirstGov.gov Search
psbot/
Sqworm/
MOSES
WebWasher
CacheBlaster/
FlashGet
wwwxref/
VoilaBot/
Gaisbot/
htdig/
grub-client
Grubclient-
SQ Webscanner
SlySearch/
Mercator-
gsa-crawler
DOF-Verify/
WebCapture
EasyDL/
Rotondo/
InternetLinkAgent/
Gigabot/
DigOut4U
Cyberdog/
ASPseek/
http://www.almaden.ibm.com/cs/crawler
eCatch/
indylabs_marius
b2w/
AnswerChase PROve
Mass Downloader/
Teradex Mapper
Lycos_Spider
RPT-HTTPClient/
WebBOT
Szukacz/
GaisBot/
targetblaster.com/
Infoseek Sidewinder/
InfoSeek Sidewinder/
Infoseek SideWinder/
DiaGem/
Steeler/
Fluffy the spider
searchhippo.com
webbandit/
webfetch/
ah-ha.com crawler
Lytranslate/
Vagabondo/
moget/
WebZIP/
Robozilla/
Oracle Ultra Search
w3m/
EmailLeach
OrangeBot
iCab/
metacarta
TurnitinBot/
DISCoFinder
InfoLink/
crawler_for_infomine
minibot
NetResearchServer/
EmailWolf
eidetica.com/spider
polybot
HomePageSearch
spider
WebFetch
AnswerBot
UdmSearch
NG/
3DE_SEARCH2
Mozilla/3.0 (compatible)
Mozilla/3.01 (compatible;)
DoCoMo/
URLBlaze
Ad Muncher
Net Vampire/
EyeNetIE
www.galaxy.com/info/crawler.html
Net Vampire/
DoCoMo/
nabot
Pita
Knowledge Engine
Webclipping.com
SSM Agent
INGRID/
Mister PiX
IAArchiver-
DISCo Pump
NPBot-
Python-urllib/
Zeus
Lachesis
lachesis
HLoader
Caddbot/
Zao/
WWW-Mechanize/
Nessus
VoilaBot
Wget/
WebReaper
LARBIN-EXPERIMENTAL efp@gmx.net
Goldfire Server
Mozilla/2.0 (compatible; NEWT ActiveX; Win32)
T-H-U-N-D-E-R-S-T-O-N-E
thatrobotsite.com
NPBot
WebSauger
Mozilla/3.0 (compatible; Indy Library)
WebGather
MFHttpScan
NutchOrg/
SiteFinder/
search.usgs.gov
TestBot
Space Bison/
Sleipnir
QuepasaCreep
MSNBOT/
pavuk/
VSE/
SearchSpider.com/
SpiderKU/
Dattatec.com-Sitios-Top
Microsoft Internet Explorer/4.40.426
WebVac
YahooSeeker/
infomine.ucr.edu
SiteSweeper
PageFetcher/
msnbot/
GetHTMLContents
Cowbot-
freshlinks.exe
Cowdog Bot
Art-Online.com
AnswerBus
MultiText/
Dattatec.com
NaverBot-
Panopy Bot
ez-robot
Scich/
Dumbot
Web Magnet
LinkWalker
TranSGeniKBot
LinkScan/
Links SQL
Google/
antibot-
Wotbox/
SpiderEngine
X-clawler
X-crawler
TygoBot
TulipChain
NetResearchServer
LinkSweeper/
Z-Add Link Checker
web-agent/
NaverBot_dloader/
mini-robot/
www.searchnear.com
Botswana
nuSearch Spider
slurp
CrawlConvera
geometabot/
kuloko-bot/
Webdup/
JoeDog/
Dillo/
Ocelli/
GPU p2p crawler
TheUsefulbot_
Unitek UniEngine/
COAST scan engine/
BlackMask.Net Search Engine
Offline Explorer/
EAH/
Spider.TerraNautic.net
Scrubby/
Check&Get
IECheck
GoForIt.com
sohu-search
NuSearch Spider
Search-Channel
NutchCVS/
LinkAlarm/
CSE HTML Validator
oBot
WISEbot/
boitho.com-dc/
EMPAS_ROBOT
Jetbot/
iVia Site Checker
findlinks/
USS-Cosmix/
mod_accessibility
ht://check/
LookBot
CheckLinks/
Html Link Validator
osis-project.jp
Seekbot/
Iltrovatore-Setaccio/
Website Downloader
BDFetch
CreativeCommons/
Nutch
SOFT411 Directory
Yandex/
ClariaBot/
MSNPTC/
heritrix/
HiSoftware AccVerify
Diamond/
OmniFind
Download Ninja
Artera
contype
FAST-WebCrawler/
Yahoo-VerticalCrawler-FormerWebCrawler/
falcon/
Shareaza
ZipppBot/
WorQmada/
Web Link Validator
CFNetwork/
CyberSpyder
FAST Data Search Document Retriever/
ImageBot
ABACHOBot
Microsoft Office Protocol Discovery
Microsoft Data Access Internet Publishing Provider Cache Manager
EmailSiphon
searchmarks
f-bot test pilot
Antro.Net
www.sygol.com
HSFT - Link Scanner
w3search
Buscaplus Robi/
adminshop.com
statcrawler
Clushbot/
Blinkx/DFS-Fetch
Header_Test_Client
fast-search-engine
WebXM
WebPix
DiamondBot
Express WebPictures
www.superbargainspace.com
archive.org_bot/
omnifind
statbot
Combine/
SpiderMan
searchmarking/
SiteXpert
ContentSmartz
gazz/
FindAnISP.com_ISP_Finder
CCGCrawl/
TravelBot/
NSDL_Search_Bot
pipeLiner/
Webinator-WBI/
KummHttp/
crawler.kpricorn.org
versus.integis.ch
search.updated.com
websearchbench
Cafi/
LinkChecker/
Microsoft_Site_Analyst/
webcrawl.net
HTTrack
Buibui
Stumbler
athenusbot
HiSoftware AccMonitor Server
MetaGloss
SURF
TREX
Validator/
HooWWWer/
www.thebananatree.org/
Spinne/
Favorites Sweeper
MJ12bot/
maxomobot/
KnowItAll
VisWeb
WebIndex/
BruinBot
ShowLinks/
Acme.Spider
Nogate
CD-Preload
websphinx.test
Missigua Locator
ApacheBench
RSSbot/
Yahoo-Newscrawler/
P3P Client
googlebot
WebAuto/
SiteSnagger
Plucker/
NetShift=
websitemirror
Eventware/
Akamai-SiteSnapshot/
zagrebin
theusefulbot
inetbot/
iSiloX/
hcat/
gnome-vfs/
didaxusbot
WebTrafficExpress/
Talkro Web-Shot/
TPSystem
SuperBot/
SpaceBison/
SiteSucker/
SEB Spider
Poodle predictor
PhpDig/
PageRank Monitor
NASA Search
MonkeyCrawl/
Mag-Net
Lament/
LMQueueBot/
K.S.Bot
Hooblybot-Image/
HTMLParser/
GoodJelly/
DiGi-RSSBot
CydralSpider/
Amfibibot/
AONDE-Spider/
moduna.com/
BecomeBot/
YottaCars_Bot/
DOY/
Forest Conservation Spider
Patwebbot
tankvit@e-mail.ru
Eco-Portal Spider
sherlock/
Knowledge.com/
smartwit.com
W3C-WebCon/
Digger/
USyd-NLP-Spider
ichiro/
Link Checker/
accoona
EduGovSearch/
WinHTTP Example/
FDM 1.x
unchaos_crawler_
TygoProwler
grub crawler
IRLbot/
GigabotSiteSearch/
FreshDownload/
WinHttp.WinHttpRequest.
ClimateArk Spider
BoaConstrictor/
CyberNavi_WebGet/
updated/
wish-la
Custo
InelaBot/
oegp v.
Water Conserve Spider
labourunions411/
SpeedySpider
FindWeb
OmiExplorer_Bot/
btbot/
Govbot/
tScholarsBot
OmniExplorer_Bot/
www.unchaos.com
NWSpider
www.nameprotect.com
cfetch/
AtlocalBot/
abot/
SCrawlTest/
Forests.org Spider
aipbot/
MojeekBot/
DataFountains/DMOZ Downloader
Speedy Spider
McBot/
BigCliqueBot/
eStyleSearch
ProjectWF-java-test-crawler
Rational SiteCheck
wgao@genieknows.com
Download Master
Kitenga-crawler-bot
BigCliqueBOT/
Ocean Conserve Spider
snap.com beta crawler
IlTrovatore-Setaccio/
Blaiz-Bee/
Metager2
GOFORITBOT
genevabot
versus crawler
Filangy/
Twiceler
Eco-Portal http://www.environmentalsustainability.info/
Yahoo-MMAudVid/
nicebot
Squid-Prefetch
parasite
cookieNET
PageBitesHyperBot/
Dir_Snatch.exe
CipinetBot
WebZIP
Arachmo
ccubee/
IEAutoDiscovery
Bilbo/
WebZIP
CE-Preload
endeca
HSFT - LVU Scanner
rssImagesBot/
Twisted PageGetter
Space Fung/
BrowserEmulator/
Microsoft Scheduled Cache Content Download Service
locust
PoCoHTTP
Web Site Downloader
Seeker.lookseek.com
Norbert the Spider
searchbot
grapeFX/
CSHttpClient/
AIBOT/
Miva
iVia/
WebIndexer/
Microsoft URL Control
AF Knowledge Now Verity Spider
MemacBot
ExactSearch
UofTDB_experiment
www.Syntryx.com
Charlotte/
NewMedhunt/
Metaspinner/
LeechGet
yacy.net
MSRBOT/
SygolBot
Zyte/
axfeedsbot/
testbot
OnetSzukaj/
LinksManager.com_bot
Helix/
MyEngines-US-Bot
maxamine.com--robot
SBIder/
Newsgroupreporter
IpselonBot/
byindia/
Xerka WebBot
Xerka MetaBot
Der groe BilderSauger
Tecomi Bot
fgcrawler
TutorGigBot/
yoono/
LocalcomBot/
dtSearchSpider
WebMiner/
wbdbot
via translate.google.com
WebDownloader for X
wume_crawler/
Eco Earth Spider
Checkbot/
Thumbnail.CZ robot
National Park Service Dan Buan (301) 213-4549
KATATUDO-Spider
xirq/
Tarantula/
its-learning crawler
Cuasarbot

# Host names
rubys-e-1-msl.cutthroatcom.net
220.114.3.9
cvn012123.bai.ne.jp
host81-130-144-116.in-addr.btopenworld.com
node-c-4e0c.a2000.nl
62.194.78.12
216.32.82.18
65.119.150.201
65.119.150.202
65.119.150.203
65.119.150.204
65.119.150.205
65.119.150.206
65.119.150.207
65.119.150.208
65.119.150.209
65.119.150.210
65.119.150.211
65.119.150.212
65.119.150.213
67.11.222.90
209.217.67.146