Bots downloading disk images – Digital Corpora
I’m preparing some statistics on who (and what) are downloading the disk images we have here at digitalcorpora.org. The first thing that I’ve done is suppress the bots that are, for whatever reason, downloading the images.
Here’s the bots that we’ve found, and the number of times each image has been downloaded by a bot.
 Rank     Count     Value(s):
  ============================
      1      2334      Mozilla/5.0 (compatible; Googlebot/2.1; +https://www.google.com/bot.html)
      2       851      MLBot (www.metadatalabs.com/mlbot)
      3       811      SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; Googlebot-Mobile/2.1; +https://www.google.com/bot.html)
      4       749      Mozilla/5.0 (compatible; DotBot/1.1; https://www.dotnetdotcom.org/, crawler@dotnetdotcom.org)
      5       492      Mozilla/5.0 (compatible; YandexBot/3.0; +https://yandex.com/bots)
      6       130      Mozilla/5.0 (compatible; bingbot/2.0; +https://www.bing.com/bingbot.htm)
      7       115      Mozilla/5.0 (compatible; DBLBot/1.0; +https://www.dontbuylists.com/)
      8       109      msnbot/2.0b (+https://search.msn.com/msnbot.htm)
      9       108      Mozilla/5.0 (compatible; SiteBot/0.1; +https://www.sitebot.org/robot/)
     10        89      CCBot/1.0 (+https://www.commoncrawl.org/bot.html)
     11        87      Mozilla/5.0 (Twiceler-0.9 https://www.cuil.com/twiceler/robot.html)
     12        78      TwengaBot-Discover (https://www.twenga.fr/bot-discover.html)
     13        58      Mozilla/5.0 (compatible; Purebot/1.1; +https://www.puritysearch.net/)
     14        51      msnbot/1.1 (+https://search.msn.com/msnbot.htm)
     15        26      Mozilla/5.0 (compatible; MJ12bot/v1.3.2; https://www.majestic12.co.uk/bot.php?+)
     16        21      Cityreview Robot (+https://www.cityreview.org/crawler/)
     17        18      'citeseerxbot'
     18        15      SindiceBot (heritrix/2.0.2 +https://sindice.com/developers/bot)
     19        12      Mozilla/5.0 (compatible; MJ12bot/v1.3.1; https://www.majestic12.co.uk/bot.php?+)
     20        11      Mozilla/5.0 (compatible; discobot/1.1; +https://discoveryengine.com/discobot.html
     21         9      Mozilla/5.0 (compatible; Exabot/3.0; +https://www.exabot.com/go/robot)
     22         7      CatchBot/3.0; +https://www.catchbot.com
                7      CyberPatrol SiteCat Webbot (https://www.cyberpatrol.com/cyberpatrolcrawler.asp)
                7      yacybot (amd64 Linux 2.6.26-2-xen-amd64; java 1.6.0_20; Europe/en) https://yacy.net/bot.html
     25         6      Mozilla/5.0 (compatible; Search17Bot/1.1; https://www.search17.com/bot.php
                6      yacybot (amd64 Linux 2.6.26-2-xen-amd64; java 1.6.0_20; Europe/de) https://yacy.net/bot.html
     27         5      MSRBOT (https://research.microsoft.com/research/sv/msrbot/)
                5      yacybot (amd64 Linux 2.6.31-20-generic; java 1.6.0_15; Europe/en) https://yacy.net/bot.html
                5      yacybot (i386 Linux 2.6.32-trunk-686; java 1.6.0_18; America/en) https://yacy.net/bot.html
     30         3      msnbot-media/1.1 (+https://search.msn.com/msnbot.htm)
     31         2      Mozilla/5.0 (SnapPreviewBot) Gecko/20061206 Firefox/1.5.0.9
                2      yacybot (amd64 Linux 2.6.26-2-amd64; java 1.6.0_20; Europe/en) https://yacy.net/bot.html
                2      yacybot (amd64 Linux 2.6.28-18-generic; java 1.6.0_19; GMT/en) https://yacy.net/bot.html
                2      yacybot (i386 Linux 2.6.31-21-generic; java 1.6.0_0; Europe/en) https://yacy.net/bot.html
     35         1      Mozilla/5.0 (compatible; Googlebot/2.1;  https://www.google.com/bot.html
                1      Mozilla/5.0 (compatible; discobot/1.1; +https://discoveryengine.com/discobot.html)
                1      findfiles.net/0.96 (Robot;test_robot@gmx-topmail.de)
                1      librabot/1.0 (+https://search.msn.com/msnbot.htm)
                1      yacybot (amd64 Linux 2.6.18-164.11.1.el5xen; java 1.6.0; Europe/en) https://yacy.net/bot.html
                1      yacybot (amd64 Linux 2.6.18-164.15.1.el5; java 1.6.0_14; Europe/de) https://yacy.net/bot.html
                1      yacybot (x86 Windows XP 5.1; java 1.6.0_18; Europe/de) https://yacy.net/bot.html
                1      yacybot (x86 Windows XP 5.1; java 1.6.0_20; Europe/de) https://yacy.net/bot.html
                1      yacybot (x86_64 Mac OS X 10.6.4; java 1.6.0_20; America/en) https://yacy.net/bot.html 

Total items printed: 6242