Repository: certtools/intelmq

Issue:
Blueliv dependendcy breaks full-test in CI
Blueliv dependendcy breaks full-test in CI:

```
+ pip install -r intelmq/bots/collectors/blueliv/REQUIREMENTS.txt
intelmq/bots/collectors/blueliv/REQUIREMENTS.txt
Collecting git+https://github.com/Blueliv/api-python-sdk (from -r intelmq/bots/collectors/blueliv/REQUIREMENTS.txt (line 4))
  Cloning https://github.com/Blueliv/api-python-sdk to /tmp/pip-req-build-k84fqhvt
  Running command git clone --filter=blob:none --quiet https://github.com/Blueliv/api-python-sdk /tmp/pip-req-build-k84fqhvt
  Resolved https://github.com/Blueliv/api-python-sdk to commit d1228945ecf44b5a93df2e51c4255aee2987ad32
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'done'
Collecting requests<=2.5.1,>=2.4.0 (from blueliv-python-sdk==2.3.0->-r intelmq/bots/collectors/blueliv/REQUIREMENTS.txt (line 4))
  Downloading requests-2.5.1-py2.py3-none-any.whl.metadata (28 kB)
Requirement already satisfied: python-dateutil>=2.4.0 in /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages (from blueliv-python-sdk==2.3.0->-r intelmq/bots/collectors/blueliv/REQUIREMENTS.txt (line 4)) (2.9.0.post0)
Requirement already satisfied: six>=1.5 in /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages (from python-dateutil>=2.4.0->blueliv-python-sdk==2.3.0->-r intelmq/bots/collectors/blueliv/REQUIREMENTS.txt (line 4)) (1.17.0)
Downloading requests-2.5.1-py2.py3-none-any.whl (464 kB)
Building wheels for collected packages: blueliv-python-sdk
  Building wheel for blueliv-python-sdk (setup.py): started
  Building wheel for blueliv-python-sdk (setup.py): finished with status 'error'
  error: subprocess-exited-with-error
  
  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [8 lines of output]
      /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/distutils/dist.py:274: UserWarning: Unknown distribution option: 'test_requires'
        warnings.warn(msg)
      usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
         or: setup.py --help [cmd1 cmd2 ...]
         or: setup.py --help-commands
         or: setup.py cmd --help
      
      error: invalid command 'bdist_wheel'
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for blueliv-python-sdk
  Running setup.py clean for blueliv-python-sdk
Failed to build blueliv-python-sdk
ERROR: Failed to build installable wheels for some pyproject.toml based projects (blueliv-python-sdk)
```

The library `blueliv-python-sdk` isn't maintained any more: https://github.com/Blueliv/api-python-sdk and their website doesn't work: https://blueliv.com so we probably have to remove it from IntelMQ.

However, it doesn't really explain the issue - do we need to install `wheel` in CI manually now? 


Discussion:
I have looked at that dependency too some weeks ago and also came to the conclusion that we need to remove the bot. It's not working anyway with the broken dependency.



Patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index f4f4cada70..093f2265fe 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -25,9 +25,11 @@ Please refer to the [NEWS](NEWS.md) for a list of changes which have an affect o
 
 ### Bots
 #### Collectors
+- Remove `intelmq.bots.collectors.blueliv` as it uses an unmaintained library, does not work any more and breaks other CI tests (fixes #2593, PR#2632 by Sebastian Wagner).
 
 #### Parsers
 - `intelmq.bots.parsers.cymru.parser_cap_program`: Add mapping for TOR and ipv6-icmp protocol (PR#2621 by Mikk Margus Möll).
+- Remove `intelmq.bots.collectors.blueliv` as it is obsolete with the removed collector (PR#2632 by Sebastian Wagner).
 
 #### Experts
 - `intelmq.bots.experts.asn_lookup.expert`:
diff --git a/docs/user/bots.md b/docs/user/bots.md
index 2fcba0ac30..f7c6719940 100644
--- a/docs/user/bots.md
+++ b/docs/user/bots.md
@@ -1045,34 +1045,6 @@ input. If you intend to link two IntelMQ instance via TCP, have a look at the TC
 
 ---
 
-### Blueliv Crimeserver <div id="intelmq.bots.collectors.blueliv.collector_crimeserver" />
-
-Collects report messages from Blueliv API.
-
-For more information visit <https://github.com/Blueliv/api-python-sdk>
-
-**Module:** `intelmq.bots.collectors.blueliv.collector_crimeserver`
-
-**Requirements**
-
-Install the required library:
-
-```bash
-pip3 install -r intelmq/bots/collectors/blueliv/REQUIREMENTS.txt
-```
-
-**Parameters (also expects [feed parameters](#feed-parameters)):**
-
-**`api_key`**
-
-(required, string) location of information resource, see <https://map.blueliv.com/?redirect=get-started#signup>
-
-**`api_url`**
-
-(optional, string) The optional API endpoint. Defaults to `https://freeapi.blueliv.com`.
-
----
-
 ### Calidog Certstream <div id="intelmq.bots.collectors.calidog.collector_certstream" />
 
 A Bot to collect data from the Certificate Transparency Log (CTL). This bot works based on certstream library
@@ -1388,16 +1360,6 @@ No additional parameters.
 
 ---
 
-### Blueliv Crimeserver <div id="intelmq.bots.parsers.blueliv.parser_crimeserver" />
-
-Parses data from Blueliv Crimeserver feed.
-
-**Module:** `intelmq.bots.parsers.blueliv.parser_crimeserver`
-
-No additional parameters.
-
----
-
 ### Calidog Certstream <div id="intelmq.bots.parsers.calidog.parser_certstream" />
 
 Parses data from Certificate Transparency Log.
diff --git a/intelmq/bots/collectors/blueliv/REQUIREMENTS.txt b/intelmq/bots/collectors/blueliv/REQUIREMENTS.txt
deleted file mode 100644
index ed4449f8ae..0000000000
--- a/intelmq/bots/collectors/blueliv/REQUIREMENTS.txt
+++ /dev/null
@@ -1,4 +0,0 @@
-# SPDX-FileCopyrightText: 2016 Sebastian Wagner
-# SPDX-License-Identifier: AGPL-3.0-or-later
-
-git+https://github.com/Blueliv/api-python-sdk
diff --git a/intelmq/bots/collectors/blueliv/__init__.py b/intelmq/bots/collectors/blueliv/__init__.py
deleted file mode 100644
index e69de29bb2..0000000000
diff --git a/intelmq/bots/collectors/blueliv/collector_crimeserver.py b/intelmq/bots/collectors/blueliv/collector_crimeserver.py
deleted file mode 100644
index ff55b0abd6..0000000000
--- a/intelmq/bots/collectors/blueliv/collector_crimeserver.py
+++ /dev/null
@@ -1,47 +0,0 @@
-# SPDX-FileCopyrightText: 2015 robcza
-#
-# SPDX-License-Identifier: AGPL-3.0-or-later
-
-# -*- coding: utf-8 -*-
-import json
-import logging
-
-from intelmq.lib.bot import CollectorBot
-from intelmq.lib.exceptions import MissingDependencyError
-
-try:
-    from sdk.blueliv_api import BluelivAPI
-except ImportError:
-    BluelivAPI = None
-
-
-class BluelivCrimeserverCollectorBot(CollectorBot):
-    """Collect reports from the Blueliv Crimeserver API"""
-    api_key: str = "<insert your api key>"
-    api_url: str = "https://freeapi.blueliv.com"
-    rate_limit: int = 3600
-
-    def init(self):
-        if BluelivAPI is None:
-            raise MissingDependencyError("sdk.blueliv_api.BluelivAPI")
-
-    def process(self):
-        self.logger.debug("Downloading report through API.")
-        proxy = None
-        if self.http_proxy and self.https_proxy:
-            proxy = {'http': self.http_proxy,
-                     'https': self.https_proxy}
-        api = BluelivAPI(base_url=self.api_url,
-                         token=self.api_key,
-                         log_level=logging.INFO,
-                         proxy=proxy)
-
-        response = api.crime_servers.online()
-        self.logger.info("Report downloaded.")
-
-        report = self.new_report()
-        report.add("raw", json.dumps([item for item in response.items]))
-        self.send_message(report)
-
-
-BOT = BluelivCrimeserverCollectorBot
diff --git a/intelmq/bots/parsers/blueliv/__init__.py b/intelmq/bots/parsers/blueliv/__init__.py
deleted file mode 100644
index e69de29bb2..0000000000
diff --git a/intelmq/bots/parsers/blueliv/parser_crimeserver.py b/intelmq/bots/parsers/blueliv/parser_crimeserver.py
deleted file mode 100644
index 0e1c287e4d..0000000000
--- a/intelmq/bots/parsers/blueliv/parser_crimeserver.py
+++ /dev/null
@@ -1,69 +0,0 @@
-# SPDX-FileCopyrightText: 2015 robcza
-#
-# SPDX-License-Identifier: AGPL-3.0-or-later
-
-# -*- coding: utf-8 -*-
-"""
-"""
-
-import json
-
-from intelmq.lib import utils
-from intelmq.lib.bot import ParserBot
-
-TYPES = {
-    'PHISHING': 'phishing',
-    'MALWARE': 'malware-distribution',
-    'EXPLOIT_KIT': 'exploit',
-    'BACKDOOR': 'system-compromise',
-    'TOR_IP': 'proxy',
-    'C_AND_C': 'c2-server'
-}
-
-
-class BluelivCrimeserverParserBot(ParserBot):
-    """Parse data from the Blueliv Crimeserver API"""
-
-    def process(self):
-        report = self.receive_message()
-
-        raw_report = utils.base64_decode(report.get('raw'))
-
-        for item in json.loads(raw_report):
-            event = self.new_event(report)
-            tor_node = False
-            if 'type' in item:
-                event.add('classification.type', TYPES[item['type']])
-                if item['type'] == 'TOR_IP':
-                    event.add('source.tor_node', True)
-                    tor_node = True
-            if 'url' in item:
-                # crimeserver reports tor ips in url as well, skip those
-
-                raise_failure = False if tor_node else True
-                event.add('source.url', item['url'],
-                          raise_failure=raise_failure)
-            if 'ip' in item:
-                event.add('source.ip', item['ip'])
-            if 'country' in item:
-                event.add('source.geolocation.cc', item['country'])
-            if 'lastSeenAt' in item:
-                event.add('time.source', item['lastSeenAt'][:-4] + '00:00')
-
-            # add some other remaining fields as extra
-            additional = {}
-            if 'status' in item:
-                additional['status'] = item['status']
-            if 'confidence' in item:
-                additional['confidence'] = item['confidence']
-            if 'updatedAt' in item:
-                additional['time_updated'] = item['updatedAt']
-            if 'firstSeenAt' in item:
-                additional['time_first_seen'] = item['firstSeenAt']
-            event.add("extra", additional)
-            event.add("raw", json.dumps(item, sort_keys=True))  # sorting for undefined order
-            self.send_message(event)
-        self.acknowledge_message()
-
-
-BOT = BluelivCrimeserverParserBot
diff --git a/intelmq/etc/feeds.yaml b/intelmq/etc/feeds.yaml
index 258e91c1f7..f132bfe65c 100644
--- a/intelmq/etc/feeds.yaml
+++ b/intelmq/etc/feeds.yaml
@@ -188,39 +188,6 @@ providers:
       revision: 2020-07-07
       documentation: https://urlhaus.abuse.ch/feeds/
       public: true
-  Blueliv:
-    CrimeServer:
-      description: Blueliv Crimeserver Collector is the bot responsible to get the
-        report through the API.
-      additional_information:
-        The service uses a different API for free users and paying subscribers. In 'CrimeServer'
-        feed the difference lies in the data points present in the feed. The non-free API
-        available from Blueliv contains, for this specific feed, following extra fields not
-        present in the free API;
-        "_id" - Internal unique ID
-        "subType" - Subtype of the Crime Server
-        "countryName" - Country name where the Crime Server is located, in English
-        "city" - City where the Crime Server is located
-        "domain" - Domain of the Crime Server
-        "host" - Host of the Crime Server
-        "createdAt" - Date when the Crime Server was added to Blueliv CrimeServer database
-        "asnCidr" - Range of IPs that belong to an ISP (registered via Autonomous System Number (ASN))
-        "asnId" - Identifier of an ISP registered via ASN
-        "asnDesc" Description of the ISP registered via ASN
-      bots:
-        collector:
-          module: intelmq.bots.collectors.blueliv.collector_crimeserver
-          parameters:
-            rate_limit: 3600
-            api_key: __APIKEY__
-            name: __FEED__
-            provider: __PROVIDER__
-        parser:
-          module: intelmq.bots.parsers.blueliv.parser_crimeserver
-          parameters:
-      revision: 2018-01-20
-      documentation: https://www.blueliv.com/
-      public: false
   Team Cymru:
     CAP:
       description: Team Cymru provides daily lists of compromised or abused devices
diff --git a/intelmq/lib/upgrades.py b/intelmq/lib/upgrades.py
index ee22b60a69..7104b47d46 100644
--- a/intelmq/lib/upgrades.py
+++ b/intelmq/lib/upgrades.py
@@ -41,7 +41,8 @@
            'v320_update_turris_greylist_url',
            'v322_url_replacement',
            'v322_removed_feeds_and_bots',
-           'v340_deprecations'
+           'v340_deprecations',
+           'v341_blueliv_removal',
            ]
 
 
@@ -974,6 +975,29 @@ def v340_deprecations(configuration, harmonization, dry_run, **kwargs):
     return message or changed, configuration, harmonization
 
 
+def v341_blueliv_removal(configuration, harmonization, dry_run, **kwargs):
+    """
+    Remove blueliv collector and parser
+    """
+    message = None
+    discontinued_bots = []
+    discontinued_bots_modules = (
+        "intelmq.bots.collectors.blueliv.collector_crimeserver",
+        "intelmq.bots.parsers.blueliv.parser_crimeserver",
+    )
+
+    for bot_id, bot in configuration.items():
+        if bot_id == 'global':
+            continue
+        if bot["module"] in discontinued_bots_modules:
+            discontinued_bots.append(bot_id)
+
+    if discontinued_bots:
+        message = f"Found discontinued bots: {', '.join(discontinued_bots)}. Remove the affected bots from the configuration."
+
+    return message, configuration, harmonization
+
+
 UPGRADES = OrderedDict([
     ((1, 0, 0, 'dev7'), (v100_dev7_modify_syntax,)),
     ((1, 1, 0), (v110_shadowserver_feednames, v110_deprecations)),
@@ -1004,7 +1028,7 @@ def v340_deprecations(configuration, harmonization, dry_run, **kwargs):
     ((3, 3, 0), ()),
     ((3, 3, 1), ()),
     ((3, 4, 0), (v340_deprecations, )),
-    ((3, 4, 1), ()),
+    ((3, 4, 1), (v341_blueliv_removal, )),
 ])
 
 ALWAYS = (harmonization,)


Test patch:
diff --git a/intelmq/tests/bots/collectors/blueliv/__init__.py b/intelmq/tests/bots/collectors/blueliv/__init__.py
deleted file mode 100644
index e69de29bb2..0000000000
diff --git a/intelmq/tests/bots/collectors/blueliv/test_collector.py b/intelmq/tests/bots/collectors/blueliv/test_collector.py
deleted file mode 100644
index f23dda633b..0000000000
--- a/intelmq/tests/bots/collectors/blueliv/test_collector.py
+++ /dev/null
@@ -1,12 +0,0 @@
-# SPDX-FileCopyrightText: 2016 Sebastian Wagner
-#
-# SPDX-License-Identifier: AGPL-3.0-or-later
-
-# -*- coding: utf-8 -*-
-"""
-Testing Blueliv crimeserver collector
-"""
-import os
-
-if os.environ.get('INTELMQ_TEST_EXOTIC'):
-    import intelmq.bots.collectors.blueliv.collector_crimeserver
diff --git a/intelmq/tests/bots/parsers/blueliv/__init__.py b/intelmq/tests/bots/parsers/blueliv/__init__.py
deleted file mode 100644
index e69de29bb2..0000000000
diff --git a/intelmq/tests/bots/parsers/blueliv/test_parser_crimeserver.data b/intelmq/tests/bots/parsers/blueliv/test_parser_crimeserver.data
deleted file mode 100644
index acce7bf392..0000000000
--- a/intelmq/tests/bots/parsers/blueliv/test_parser_crimeserver.data
+++ /dev/null
@@ -1,5 +0,0 @@
-[
-{"confidence": 4, "country": "US", "firstSeenAt": "2015-10-20T06:14:00+0000", "ip": "199.34.228.54", "lastSeenAt": "2015-12-09T04:43:29+0000", "latitude": 37.7989, "longitude": -122.3984, "status": "ONLINE", "type": "MALWARE", "updatedAt": "2015-12-09T04:45:53+0000", "url": "http://amirosterweil.weebly.com/uploads/2/3/2/5/23258576/corporate_elearning.pdf"},
-{"country": "US", "firstSeenAt": "2015-07-04T17:08:23+0000", "lastSeenAt": "2015-12-13T13:53:55+0000", "latitude": 33.6119, "longitude": -111.8906, "status": "ONLINE", "type": "PHISHING", "updatedAt": "2015-12-13T13:55:54+0000", "url": "http://mondeos-italo.com/store/apple/"},
-{"ip": "10.0.0.1", "lastSeenAt": "2015-12-13T17:26:23+0000", "latitude": 33.749, "longitude": -84.388, "status": "ONLINE", "type": "TOR_IP", "url": "10.0.0.1"}
-]
diff --git a/intelmq/tests/bots/parsers/blueliv/test_parser_crimeserver.data.license b/intelmq/tests/bots/parsers/blueliv/test_parser_crimeserver.data.license
deleted file mode 100644
index a4d56fe7ad..0000000000
--- a/intelmq/tests/bots/parsers/blueliv/test_parser_crimeserver.data.license
+++ /dev/null
@@ -1,2 +0,0 @@
-SPDX-FileCopyrightText: 2015 robcza
-SPDX-License-Identifier: AGPL-3.0-or-later
diff --git a/intelmq/tests/bots/parsers/blueliv/test_parser_crimeserver.py b/intelmq/tests/bots/parsers/blueliv/test_parser_crimeserver.py
deleted file mode 100644
index cc6bb6643e..0000000000
--- a/intelmq/tests/bots/parsers/blueliv/test_parser_crimeserver.py
+++ /dev/null
@@ -1,82 +0,0 @@
-# SPDX-FileCopyrightText: 2015 robcza
-#
-# SPDX-License-Identifier: AGPL-3.0-or-later
-
-# -*- coding: utf-8 -*-
-
-import os
-import unittest
-
-import intelmq.lib.test as test
-import intelmq.lib.utils as utils
-from intelmq.bots.parsers.blueliv.parser_crimeserver import \
-    BluelivCrimeserverParserBot
-
-with open(os.path.join(os.path.dirname(__file__),
-                       'test_parser_crimeserver.data')) as handle:
-    EXAMPLE_FILE = handle.read()
-EXAMPLE_LINES = EXAMPLE_FILE.splitlines()[1:-1]
-
-EXAMPLE_REPORT = {"feed.name": "Blueliv Crimeserver",
-                  "raw": utils.base64_encode(EXAMPLE_FILE),
-                  "__type": "Report",
-                  "time.observation": "2015-09-02T14:17:58+00:00"
-                  }
-EXAMPLE_EVENTS = [{
-    "__type": "Event",
-    "feed.name": "Blueliv Crimeserver",
-    "classification.type": "malware-distribution",
-    "raw": utils.base64_encode(EXAMPLE_LINES[0][:-1]),
-    "extra.confidence": 4,
-    "extra.time_updated": "2015-12-09T04:45:53+0000",
-    "extra.time_first_seen": "2015-10-20T06:14:00+0000",
-    "extra.status": "ONLINE",
-    "source.url": "http://amirosterweil.weebly.com/uploads/2/3/2/5/23258576/corporate_elearning.pdf",
-    "time.source": "2015-12-09T04:43:29+00:00",
-    "time.observation": "2015-09-02T14:17:58+00:00",
-    "source.geolocation.cc": "US",
-    "source.ip": "199.34.228.54"
-    }, {
-    "__type": "Event",
-    "feed.name": "Blueliv Crimeserver",
-    "classification.type": "phishing",
-    "raw": utils.base64_encode(EXAMPLE_LINES[1][:-1]),
-    "extra.status": "ONLINE",
-    "extra.time_updated": "2015-12-13T13:55:54+0000",
-    "extra.time_first_seen": "2015-07-04T17:08:23+0000",
-    "source.url": "http://mondeos-italo.com/store/apple/",
-    "time.source": "2015-12-13T13:53:55+00:00",
-    "time.observation": "2015-09-02T14:17:58+00:00",
-    "source.geolocation.cc": "US",
-    }, {
-    "__type": "Event",
-    "feed.name": "Blueliv Crimeserver",
-    "extra.status": "ONLINE",
-    "classification.type": "proxy",
-    "raw": utils.base64_encode(EXAMPLE_LINES[2]),
-    "source.tor_node": True,
-    "time.source": "2015-12-13T17:26:23+00:00",
-    "time.observation": "2015-12-13T17:26:23+00:00",
-    "source.ip": "10.0.0.1"
-}]
-
-
-class TestBluelivCrimeserverParserBot(test.BotTestCase, unittest.TestCase):
-    """
-    A TestCase for BluelivCrimeserverParserBot.
-    """
-
-    @classmethod
-    def set_bot(cls):
-        cls.bot_reference = BluelivCrimeserverParserBot
-        cls.default_input_message = EXAMPLE_REPORT
-
-    def test_event(self):
-        """ Test if correct Event has been produced. """
-        self.run_bot()
-        self.assertMessageEqual(0, EXAMPLE_EVENTS[0])
-        self.assertMessageEqual(1, EXAMPLE_EVENTS[1])
-        self.assertMessageEqual(2, EXAMPLE_EVENTS[2])
-
-if __name__ == '__main__':  # pragma: no cover
-    unittest.main()
diff --git a/intelmq/tests/lib/test_upgrades.py b/intelmq/tests/lib/test_upgrades.py
index a30800b9cb..11805e8bc2 100644
--- a/intelmq/tests/lib/test_upgrades.py
+++ b/intelmq/tests/lib/test_upgrades.py
@@ -616,6 +616,15 @@
         "module": "intelmq.bots.collectors.twitter.collector",
     },
 }
+V341_BLUELIV_REMOVAL = {
+    "global": {},
+    "blueliv-collector": {
+        "module": "intelmq.bots.collectors.blueliv.collector_crimeserver"
+    },
+    "blueliv-parser": {
+        "module": "intelmq.bots.parsers.blueliv.parser_crimeserver"
+    }
+}
 
 
 
@@ -856,6 +865,12 @@ def test_v340_twitter_collector(self):
         self.assertIn('twitter-collector', result[0])
         self.assertEqual(V340_TWITTER_COLLECTOR_IN, result[1])
 
+    def test_v341_blueliv_removal(self):
+        """ Test v341_blueliv_removal deprecation warning """
+        result = upgrades.v341_blueliv_removal(V341_BLUELIV_REMOVAL, {}, False)
+        self.assertIn('blueliv-collector', result[0])
+        self.assertIn('blueliv-parser', result[0])
+        self.assertEqual(V341_BLUELIV_REMOVAL, result[1])
 
 for name in upgrades.__all__:
     setattr(TestUpgradeLib, 'test_function_%s' % name,