Repository: sopel-irc/sopel

Issue:
safety: Malware Domains is dead
### Description
A cybersec firm bought Malware Domains and shut down its information feeds as of November 30, 2020.

Partly, this means that `safety` no longer works. But it _does_ support checking against VirusTotal, right? (If one acquires an API key.) Problem is that `safety` won't even _load_ now that it can't fetch Malware Domains' list.

### Reproduction steps
1. Start Sopel with the `safety` plugin enabled (which it is by default)
2. `safety` throws an exception during setup and fails to load

### Expected behavior
`safety` at least continues to operate using the last-fetched `malwaredomains.txt` and/or VirusTotal API access (if configured).

### Logs
```
[2020-12-22 13:35:56,136] sopel.modules.safety INFO     - Downloading malwaredomains db from https://mirror1.malwaredomains.com/files/justdomains
[2020-12-22 13:35:56,332] sopel.bot            ERROR    - Error in safety setup: HTTP Error 404: Not Found
Traceback (most recent call last):
  File "/mnt/c/Users/dgw/github/sopel/sopel/bot.py", line 310, in setup_plugins
    plugin.setup(self)
  File "/mnt/c/Users/dgw/github/sopel/sopel/plugins/handlers.py", line 265, in setup
    self._module.setup(bot)
  File "/mnt/c/Users/dgw/github/sopel/sopel/modules/safety.py", line 98, in setup
    _download_malwaredomains_db(loc)
  File "/mnt/c/Users/dgw/github/sopel/sopel/modules/safety.py", line 116, in _download_malwaredomains_db
    urlretrieve(url, path)
  File "/usr/lib/python3.6/urllib/request.py", line 248, in urlretrieve
    with contextlib.closing(urlopen(url, data)) as fp:
  File "/usr/lib/python3.6/urllib/request.py", line 223, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.6/urllib/request.py", line 532, in open
    response = meth(req, response)
  File "/usr/lib/python3.6/urllib/request.py", line 642, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python3.6/urllib/request.py", line 570, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.6/urllib/request.py", line 504, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.6/urllib/request.py", line 650, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 404: Not Found
```

### Environment
- Sopel `.version`: `master` or 7.0.x
- Sopel installed via: `pip` (`-e` or from PyPI)
- Python version: 3.6.9 or 3.8.2
- Operating system: Any
- Relevant plugins: safety

### Notes
~~I created a [question](https://security.stackexchange.com/q/242453/601) on the Security Stack Exchange asking for Malware Domains replacement options.~~ Stack Exchange is so useless; they close just about everything as off-topic 😡

At minimum, this issue should be resolved by a PR that removes `malwaredomains.txt` fetching. Ideally that PR would replace it with an equivalent service, and add error handling/logging for if fetching the new list fails.

If Malware Domains cannot be replaced, and VirusTotal's API becomes the only option, remove the `safety` plugin to its own package (core plugins MUST NOT require an API key to function).


Discussion:
There are a couple of candidates on https://github.com/StevenBlack/hosts and on https://firebog.net/ ("Malicious")

Resolving patch:
diff --git a/sopel/modules/safety.py b/sopel/modules/safety.py
--- a/sopel/modules/safety.py
+++ b/sopel/modules/safety.py
@@ -77,7 +77,7 @@ def configure(config):
     config.safety.configure_setting(
         'vt_api_key',
         "Optionally, enter a VirusTotal API key to improve malicious URL "
-        "protection.\nOtherwise, only the Malwarebytes DB will be used."
+        "protection.\nOtherwise, only the StevenBlack list will be used."
     )
 
 
@@ -91,18 +91,40 @@ def setup(bot):
     for item in bot.config.safety.known_good:
         known_good.append(re.compile(item, re.I))
 
-    loc = os.path.join(bot.config.homedir, 'malwaredomains.txt')
+    old_file = os.path.join(bot.config.homedir, 'malwaredomains.txt')
+    if os.path.exists(old_file) and os.path.isfile(old_file):
+        LOGGER.info('Removing old malwaredomains file from %s', old_file)
+        try:
+            os.remove(old_file)
+        except Exception as err:
+            # for lack of a more specific error type...
+            # Python on Windows throws an exception if the file is in use
+            LOGGER.info('Could not delete %s: %s', old_file, str(err))
+
+    loc = os.path.join(bot.config.homedir, 'unsafedomains.txt')
     if os.path.isfile(loc):
-        if os.path.getmtime(loc) < time.time() - 24 * 60 * 60 * 7:
-            # File exists but older than one week — update it
-            _download_malwaredomains_db(loc)
+        if os.path.getmtime(loc) < time.time() - 24 * 60 * 60:
+            # File exists but older than one day — update it
+            _download_domain_list(loc)
     else:
-        _download_malwaredomains_db(loc)
+        _download_domain_list(loc)
     with open(loc, 'r') as f:
         for line in f:
             clean_line = unicode(line).strip().lower()
-            if clean_line != '':
-                malware_domains.add(clean_line)
+            if not clean_line or clean_line[0] == '#':
+                # blank line or comment
+                continue
+
+            parts = clean_line.split(' ', 1)
+            try:
+                domain = parts[1]
+            except IndexError:
+                # line does not contain a hosts entry; skip it
+                continue
+
+            if '.' in domain:
+                # only publicly routable domains matter; skip loopback/link-local stuff
+                malware_domains.add(domain)
 
 
 def shutdown(bot):
@@ -110,9 +132,9 @@ def shutdown(bot):
     bot.memory.pop('safety_cache_lock', None)
 
 
-def _download_malwaredomains_db(path):
-    url = 'https://mirror1.malwaredomains.com/files/justdomains'
-    LOGGER.info('Downloading malwaredomains db from %s', url)
+def _download_domain_list(path):
+    url = 'https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts'
+    LOGGER.info('Downloading malicious domain list from %s', url)
     urlretrieve(url, path)
 
 
@@ -180,19 +202,17 @@ def url_handler(bot, trigger):
             positives = result.get('positives', 0)
             total = result.get('total', 0)
     except requests.exceptions.RequestException:
-        # Ignoring exceptions with VT so MalwareDomains will always work
+        # Ignoring exceptions with VT so domain list will always work
         LOGGER.debug('[VirusTotal] Error obtaining response.', exc_info=True)
     except InvalidJSONResponse:
-        # Ignoring exceptions with VT so MalwareDomains will always work
+        # Ignoring exceptions with VT so domain list will always work
         LOGGER.debug('[VirusTotal] Malformed response (invalid JSON).', exc_info=True)
 
     if unicode(netloc).lower() in malware_domains:
-        # malwaredomains is more trustworthy than some VT engines
-        # therefore it gets a weight of 10 engines when calculating confidence
-        positives += 10
-        total += 10
+        positives += 1
+        total += 1
 
-    if positives > 1:
+    if positives >= 1:
         # Possibly malicious URL detected!
         confidence = '{}%'.format(round((positives / total) * 100))
         msg = (


Test patch: