Repository: cowrie/cowrie

Issue:
Add output plugin: AbuseIP Database
Similar to SANS DShield and CSIRTG, AbuseIP also offers a public list of bad actors, with ability to export feeds to FAIL2BAN.

API documentation at https://docs.abuseipdb.com/


Discussion:
@HQuest What would you like exactly that the IP's bruteforcing the honeypots are getting reported to the AbuseIP database?
I would like Cowrie to publish such attackers, so systems outside honeypots can make use of this information and apply mitigation controls against known bad actors.

> On Dec 5, 2019, at 08:37, RobertH1993 <notifications@github.com> wrote:
> 
> ﻿
> @HQuest What would you like exactly that the IP's bruteforcing the honeypots are getting reported to the AbuseIP database?
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub, or unsubscribe.

Going to work on this, when ready i will submit a PR.


@RobertH1993 Do you still still plan on working on this? I might be able to take a look at this sometime in the next couple of weeks while confined if you're no longer interested.

@HQuest Do you have a preliminary set of criteria in mind for evaluating whether to report an IP or not? I.e. do you want to report after a certain number of failed login attempts, an SFTP/Wget event, a combinations of events, etc?
I'm sorry I don't have time to work on it. If you want to please do so :)

Op di 7 apr. 2020 19:30 schreef Benjamin Stephens <notifications@github.com
>:

> @RobertH1993 <https://github.com/RobertH1993> Do you still still plan on
> working on this? I might be able to take a look at this sometime in the
> next couple of weeks while confined if you're no longer interested.
>
> @HQuest <https://github.com/HQuest> Do you have a preliminary set of
> criteria in mind for evaluating whether to report an IP or not? I.e. do you
> want to report after a certain number of failed login attempts, an
> SFTP/Wget event, a combinations of events, etc?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/cowrie/cowrie/issues/1271#issuecomment-610518533>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AF24ZELUJRCUVJXR6HQ7IOTRLNPLRANCNFSM4JOH6SOQ>
> .
>

I think a customizable count and frequency combination would be great. While one in a day could be an accident, five in a 30 second interval feels more like a brute force attempt, but it all varies on each place’s policies.

Also, anything that does a file transfer (wget/curl/etc) - nobody should be moving files without permissions- would too trigger an alarm.

That’s how I view it on a perfect world. Anything actually will be great.

Thank you for stepping up.

Alex

> On Apr 7, 2020, at 13:30, Benjamin Stephens <notifications@github.com> wrote:
> 
> ﻿
> @RobertH1993 Do you still still plan on working on this? I might be able to take a look at this sometime in the next couple of weeks while confined if you're no longer interested.
> 
> @HQuest Do you have a preliminary set of criteria in mind for evaluating whether to report an IP or not? I.e. do you want to report after a certain number of failed login attempts, an SFTP/Wget event, a combinations of events, etc?
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub, or unsubscribe.

  
  
  
  
Alright, I'll try and take a look at it in the next week and see what I can come up with. For the moment I might start with simply observing brute force login attempts and reporting any IP when it exceeds its threshold.
  

  
As for detecting brute force, I'll make the threshold settings (both time and number of allowed attempts) adjustable from the config file, but personally I'd recommend setting the threshold for attempts higher than 5 (just in case someone really has just mistyped an IP address and actually thinks they're logging into their own server, even if this is really really unlikely) and, more importantly, a time longer than thirty seconds. A behavious I've observed is bots which, I assume, rotate through a long list of IP addresses, making one guess on each one before attempting on the next. Given a window of a few minutes it might not look like much, but viewed over a long period it's quite clear that it's a brute force attack--just a very slow one.
  

  
I've also just sent a message asking the maintainers how they feel about re-reporting IPs less than 24 hours after the initial report if it's for another category of behaviour. Depending on their response and if all goes well getting an initial plugin in place, we can then look at other behaviours that could be reported. Have a think about what kinds of events you've seen in your log files that could be used as reporting criteria--things like cowrie.direct-tcpip.data, cowrie.command.input, etc--and what [AbuseIPDB reporting categories](https://www.abuseipdb.com/categories) they fit in to.
  
  

  

  

  
  
  
  
  
>   
> Activé 7 avr. 2020 à 20:55, HQuest  <notifications@github.com>  a écrit :
>   
>  I think a customizable count and frequency combination would be great. While one in a day could be an accident, five in a 30 second interval feels more like a brute force attempt, but it all varies on each place’s policies.
>   
>  Also, anything that does a file transfer (wget/curl/etc) - nobody should be moving files without permissions- would too trigger an alarm.
>   
>  That’s how I view it on a perfect world. Anything actually will be great.
>   
>  Thank you for stepping up.
>   
>  Alex
>   
>   >  On Apr 7, 2020, at 13:30, Benjamin Stephens  <notifications@github.com>  wrote:
>   >
>   >  ﻿
>   >  @RobertH1993 Do you still still plan on working on this? I might be able to take a look at this sometime in the next couple of weeks while confined if you're no longer interested.
>   >
>   >  @HQuest Do you have a preliminary set of criteria in mind for evaluating whether to report an IP or not? I.e. do you want to report after a certain number of failed login attempts, an SFTP/Wget event, a combinations of events, etc?
>   >
>   >  —
>   >  You are receiving this because you were mentioned.
>   >  Reply to this email directly, view it on GitHub, or unsubscribe.
>   
>
> —
>  You are receiving this because you commented.
>  Reply to this email directly,   view it on GitHub, or   unsubscribe.
>
>   
     
To keep you updated, I've made a start on this today. [WIP is here](https://github.com/bn-s/cowrie-abuseipdb-api-plugin/)
Really appreciated. And I loved the observer functions ;)

> On Apr 23, 2020, at 1:26 PM, Benjamin Stephens <notifications@github.com> wrote:
> 
> 
> To keep you updated, I've made a start on this today. WIP is here <https://github.com/bn-s/cowrie-abuseipdb-api-plugin/>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub <https://github.com/cowrie/cowrie/issues/1271#issuecomment-618533492>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAIJWNMZQMZT5ETARVINYJTROB24JANCNFSM4JOH6SOQ>.
> 


Please also look at the GreyNoise, DShield and Virustotal plugins. They do approximately the same.
Try to use `treq` or `twisted.web` to do your requests so they are asynchronous.
Use the standard config file, as the other plugins do, the system is extensible.
Try to put all the code in a single file, in the existing `src/cowrie/output` directory structure.
Once it's working, please submit a PR and I'll be happy to merge it.

Thanks for taking a look and for the feedback.

Just to make sure we're on the same page;

- The abuseipdb folder is just a standard plugin module to be dropped into the `src/cowrie/output` folder like any other plugin. Being a folder rather than a single file is the only difference here.
- The plugin, whilst having its own config file, does use the standard config files. Indeed, any setting that appears in the standard config files takes priority over anything in the plugin's config file. The idea here is to avoid hard-coding anything so we can make some things customisable by a user who might not want to 'fiddle around in the code', but without dropping too many options into the standard cowrie files by default. That way if, for example, a user wants to set a daily limit on reports because they also use their api quota across other honeypots/projects/etc, the option is there if they dig but, for the user who just wants to get up and running, they just need to set their api key and that's it, without the anxiety of 'do I need to think about the settings of this thing...' when really, most will not. Of course, as above, I can easily change this if you prefer I don't do it.
- One of the plugins you've mentioned above is just making plain old synchronnous requests using the standard python requests library. They're threaded so non-blocking sure, but that is where I was going if your main concern was about blocking. Otherwise, Treq looks like it doesn't take much work to put in place, so I'm happy to take a look at that soon.

If you do desire that the plugin be reduced to a single file, that leads me to a question as I was intending to dump/load a working dictionary into/from the plugin's folder on stop/start: is there another folder in the standard cowrie structure where you would be happy for me to dump a file to?
Hi! Thanks for the detailed comments!
* As long as it'll work the same as the other plugins no objection against multiple files.
* There is already a 'default options' feature in Cowrie that can be overridden. The `cowrie.cfg.dist` contains the distribution options, which can be individually overridden by entries in `cowrie.cfg`. If you are adapting an existing library with an existing config file mechanism we can talk about this, but if you are implementing a settings overriding mechanism from scratch, you are better off using existing features.
* Yes, I have accepted synchronous requests libraries merge requests in the past. They don't seem to cause a lot of issues, but the general recommendation is to write them asynchronously, this will fit better in the Twisted model. 

* There is currently no directory to store output to, but there is a `state_path` entry defined in `cowrie.cfg.dist`, so I'd use that, maybe followed by your plugin-name. By default that would then be `var/lib/cowrie/abuseip/`. I don't know how much state you're going to keep, but if it's just entries that need to be submitted, you can do those from the (now-working) `stop()` function as well. 


Okay, so the utils module looks quite long which may have given the impression that I'm up to something weird, but it's mostly just comments for myself and other utility type stuff that has nothing to do with parsing configs... like making a time string into a tuple and, later, a funciton for calculating the number of seconds into the future this time will next occur so I have something I can give to twisted reactor in a `callLater()`... There's very little in there to do with reading and overriding configs or anything like that.

Reducing it to just the bits that load configs, we're left with a vanilla configparser read and a one line dictionary merge, which is where the entirety of the custom settings overriding solution takes plae.

```python
from configparser import ConfigParser
from cowrie... import CowrieConfig

PLUGIN_CFG_FILE = file.cfg

class Confs:
    def __init__(self):
        self.__dict__.update(
            {**self.get_defaults(), **CowrieConfig()['abuseip']}
        )

    def get_defaults(self):
        parser = ConfigParser()
        f = PLUGIN_CFG_FILE
        parser.read(f)
        return parser['abuseip']
```

Seeing it extracted/reduced like so, how do you feel about it?

As for asyc requests--consider it done. I'm going to try and do a little work on the plugin later this afternoon or tomorrow morning, so I might even start there.

Hi! The standard `configparser` library has a lot of things you are reimplementing, like `configparser.getboolean` where you wrote `trudat`. There's `getint` as well, which has a fallback option in case no value is define. The merge function is already in `configparser` as well, and it's what I use to merge `cowrie.cfg` with `cowrie.cfg.dist`, that could simplify a lot of your util.py.
I still don't see why you need an additional config file, while all other plugins use `cowrie.cfg.dist`.


Okay I'll drop the config file. It's a non-essential extra so I'm okay to do without it.

For the curious readers the reason for why I did it as such is that (after making the decision to introduce an additional config file) I felt that a plugin should remain as self-contained as possible--anything the plugin does should not touch anything outside of itself unless absolutely unavoidable. So sure, I could have done the whole thing with `CowrieConfig`, but I thought that would be more controversial as it's a singleton so we don't get our own instance to play with.

So, given that I decided to include a config file, I felt the most direct and efficient solution was to just do the read internally in the plugin then merge the two dictionaries. I could have bypassed the `CowrieConfig` class and passed the file directly to the read function from the cowrie config module, but I wasn't planning the use of environement variables so really, it doesn't make too much difference either way as it's all just `configparser` and we still haven't merged the two configs. I could have also re-read the Cowrie configs using `configparser` and let my own instance sort it all out, but why perform an operation that has already been done?

So as everything is now self contained, I felt it would more efficient to just update the class dictionary with the two merged configs--we've given up a little inbuilt `configparser` functionality, but we also save lines of code and made the whole import operation far more lightweight. We never have to directly address anything that's going to remain a string and it's also more lightweight than penetrating 400 calls deep into the `configparser` labyrinth performing a get on each single item. In doing it like this we've only really introduced the risk that a user might change parameter names in the configs or try to reproduce class methods as config parameters for kicks... but this has effectively the same consequences as someone typing `Flase` as a boolean value; we get an error, the plugin doesn't start and life goes on as usual for the rest of Cowrie as we've kept the behaviour of the plugin contained to itself.

As such, yeah, I did reproduce the boolean function, but aside from that no other functionality had to be reproduced and the additional code is offset by no longer importing strings individually. Calling standard python `int()` isn't really reproducing anything; sure it's redundant if we do a `configparser` get, only to later do the conversion, but that never happened. It also has the nice side effect that readability is improved a little...

```python
# This...:
self.something = int(self.something)
self.otherthing = makebool(self.otherthing)
# ...is a bit cleaner than this:
self.something = ConfigsClass().getint('output_pluginname', 'something', fallback=1337)
self.otherthing = ConfigsClass().getboolean('output_pluginname', 'otherthing', fallback=False)
```

As for the existence of a configs class--taking the modular approach I was going to take, it made more sense to import everything and do all the converting of types and data structures in one place. Like so, everything is ready to be dropped into any module/class anywhere else, ready to go, simply by typing `cfg.parameter_name`. It just makes life a little simpler and helps make things more explicit later on.

That sums up whys of what I've done.

I've had the time this afternoon to refactor some of what I've already done and add in Treq for requests. [There's an alpha version available here](https://github.com/bn-s/cowrie-abuseipdb-api-plugin/tree/master/Alpha).

At the moment it should work... sort of... I hope. But it's completely untested so expect errors. If you do test it, please post any errors here and I'll look into them before I do a pull request.
I've made a couple of changes, done some testing, and everything is working as expected.

I've spun up a live box (68.183.65.134:22) with [the current version](https://github.com/bn-s/cowrie-abuseipdb-api-plugin/tree/master/Alpha) which is [reporting here](https://www.abuseipdb.com/user/42261). I'm going to let it run for the weekend and test a couple of things such as if AbuseIPDB are sending the Retry-After header, and if we're receiving it reliably.

If all goes well I'll submit a pull request in the new week.

Resolving patch:
diff --git a/src/cowrie/output/abuseipdb.py b/src/cowrie/output/abuseipdb.py
new file mode 100644
--- /dev/null
+++ b/src/cowrie/output/abuseipdb.py
@@ -0,0 +1,487 @@
+# MIT License                                                                 #
+#                                                                             #
+# Copyright (c) 2020 Benjamin Stephens <premier_contact@ben-stephens.net>     #
+#                                                                             #
+# Permission is hereby granted, free of charge, to any person obtaining a     #
+# copy of this software and associated documentation files (the "Software"),  #
+# to deal in the Software without restriction, including without limitation   #
+# the rights to use, copy, modify, merge, publish, distribute, sublicense,    #
+# and/or sell copies of the Software, and to permit persons to whom the       #
+# Software is furnished to do so, subject to the following conditions:        #
+#                                                                             #
+# The above copyright notice and this permission notice shall be included in  #
+# all copies or substantial portions of the Software.                         #
+#                                                                             #
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR  #
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,    #
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE #
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER      #
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING     #
+# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER         #
+# DEALINGS IN THE SOFTWARE.                                                   #
+
+
+""" Cowrie plugin for reporting login attempts via the AbuseIPDB API.
+
+"AbuseIPDB is a project dedicated to helping combat the spread of hackers,
+spammers, and abusive activity on the internet." <https://www.abuseipdb.com/>
+"""
+
+
+__author__ = 'Benjamin Stephens'
+__version__ = '0.3b3'
+
+
+import pickle
+from collections import deque
+from datetime import datetime
+from json.decoder import JSONDecodeError
+from pathlib import Path
+from sys import version_info
+from time import sleep, time
+
+from treq import post
+
+from twisted.internet import defer, reactor, threads
+from twisted.python import log
+from twisted.web import http
+
+from cowrie.core import output
+from cowrie.core.config import CowrieConfig
+
+
+# How often we clean and dump and our lists/dict...
+CLEAN_DUMP_SCHED = 600
+# ...and the file we dump to.
+DUMP_FILE = 'aipdb.dump'
+
+ABUSEIP_URL = 'https://api.abuseipdb.com/api/v2/report'
+# AbuseIPDB will just 429 us if we report an IP too often; currently 15 minutes
+# (900 seconds); set lower limit here to protect againt bad user input.
+REREPORT_MINIMUM = 900
+
+
+class Output(output.Output):
+    def start(self):
+        self.tolerance_attempts = CowrieConfig().getint('output_abuseipdb', 'tolerance_attempts', fallback=10)
+        self.state_path = CowrieConfig().get('output_abuseipdb', 'dump_path')
+        self.state_path = Path(*(d for d in self.state_path.split('/')))
+        self.state_dump = self.state_path / DUMP_FILE
+
+        if version_info.minor < 6:
+            # PathLike object not compatible with with open in python < 3.6
+            self.state_dump = str(self.state_dump)
+
+        self.logbook = LogBook(self.tolerance_attempts, self.state_dump)
+        # Pass our instance of LogBook() to Reporter() so we don't end up
+        # working with different records.
+        self.reporter = Reporter(self.logbook, self.tolerance_attempts)
+
+        # We store the LogBook state any time a shutdown occurs. The rest of
+        # our start-up is just for loading and cleaning the previous state
+        try:
+            with open(self.state_dump, 'rb') as f:
+                self.logbook.update(pickle.load(f))
+
+            # Check to see if we're still asleep after receiving a Retry-After
+            # header in a previous response
+            if self.logbook['sleeping']:
+                t_wake = self.logbook['sleep_until']
+                t_now = time()
+                if t_wake > t_now:
+                    # If we're meant to be asleep, we'll set logbook.sleep to
+                    # true and logbook.sleep_until to the time we can wake-up
+                    self.logbook.sleeping = True
+                    self.logbook.sleep_until = t_wake
+                    # and we set an alarm so the reactor knows when he can drag
+                    # us back out of bed
+                    reactor.callLater(t_wake - t_now, self.logbook.wakeup)
+
+            del self.logbook['sleeping']
+            del self.logbook['sleep_until']
+            tolerated = self.logbook.pop('tolerated')
+
+        except (pickle.UnpicklingError, FileNotFoundError, KeyError):
+            if self.state_path.exists():
+                pass
+            else:
+                # If we don't already have an abuseipdb directory, let's make
+                # one with the necessary permissions now.
+                Path(self.state_path).mkdir(mode=0o700, parents=False, exist_ok=False)
+
+        # And we do a clean-up to make sure that we're not carrying any expired
+        # entries. The clean-up task ends by calling itself in a callLater,
+        # thus running every CLEAN_DUMP_SCHED seconds until the end of time.
+        self.logbook.cleanup_and_dump_state()
+
+        # If tolerance_attempts > the previous setting, we need to change the
+        # maximum length of the deque for any previously seen IP that we're
+        # loading, otherwise we'd potentially have IPs that may never trigger
+        # a report
+        try:
+            if tolerated != self.tolerance_attempts:
+                for k in self.logbook:
+                    if self.logbook[k].__class__() == deque():
+                        self.logbook[k] = deque([*self.logbook[k]], maxlen=self.tolerance_attempts)
+        except UnboundLocalError:
+            pass
+
+        log.msg(
+            eventid='cowrie.abuseipdb.started',
+            format='AbuseIPDB Plugin version {} started. Currently in beta.'.format(__version__),
+        )
+
+    def stop(self):
+        self.logbook.cleanup_and_dump_state(mode=1)
+
+    def write(self, ev):
+        if self.logbook.sleeping:
+            return
+
+        if ev['eventid'].rsplit('.', 1)[0] == 'cowrie.login':
+            # If tolerance_attempts was set to 1 or 0, we don't need to
+            # keep logs so our handling of the event is different than if > 1
+            if self.tolerance_attempts <= 1:
+                self.intolerant_observer(ev['src_ip'], time(), ev['username'])
+            else:
+                self.tolerant_observer(ev['src_ip'], time())
+
+    def intolerant_observer(self, ip, t, uname):
+        # Checks if already reported; if yes, checks if we can rereport yet.
+        # The entry for a reported IP is a tuple (None, time_reported). If IP
+        # is not already in logbook, reports it immediately
+        if ip in self.logbook:
+            if self.logbook.can_rereport(ip, t):
+                self.reporter.report_ip_single(ip, t, uname)
+            else:
+                return
+        else:
+            self.reporter.report_ip_single(ip, t, uname)
+
+    def tolerant_observer(self, ip, t):
+        # Appends the time an IP was seen to it's list in logbook. Once the
+        # length of the list equals tolerance_attempts, the IP is reported.
+        if ip in self.logbook:
+            try:
+                if self.logbook[ip][0]:
+                    # Evaluates true if IP not already reported. If reported,
+                    # logbook entry is of the form (None, time_reported).
+                    self.logbook[ip].append(t)
+                    self.logbook.clean_expired_timestamps(ip, t)
+
+                    if len(self.logbook[ip]) >= self.tolerance_attempts:
+                        self.reporter.report_ip_multiple(ip)
+
+                elif self.logbook.can_rereport(ip, t):
+                    # Check if reported IP is ready for re-reporting
+                    self.logbook[ip] = deque([t], maxlen=self.tolerance_attempts)
+
+                else:
+                    return
+
+            except IndexError:
+                # If IP address was in logbook but had no entries then we're
+                # fine to re-report.
+                self.logbook[ip].append(t)
+
+        else:
+            self.logbook[ip] = deque([t], maxlen=self.tolerance_attempts)
+
+
+class LogBook(dict):
+    """ Dictionary class with methods for cleaning and dumping its state.
+
+    This class should be treated as global state. For the moment this is
+    achieved simply by passing the instance created by Output() directly to
+    Reporter(). Sharing is caring.
+    """
+    def __init__(self, tolerance_attempts, state_dump):
+        self.sleeping = False
+        self.sleep_until = 0
+        self.tolerance_attempts = tolerance_attempts
+        self.tolerance_window = 60 * CowrieConfig().getint('output_abuseipdb', 'tolerance_window', fallback=120)
+        self.rereport_after = 3600 * CowrieConfig().getfloat('output_abuseipdb', 'rereport_after', fallback=24)
+        if self.rereport_after < REREPORT_MINIMUM:
+            self.rereport_after = REREPORT_MINIMUM
+        self.state_dump = state_dump
+        # To write our dump to disk we have a method we call in a thread so we
+        # don't block if we get slow io. This is a cheap hack to get a lock on
+        # the file. See self.write_dump_file()
+        self._writing = False
+        super().__init__()
+
+    def wakeup(self):
+        # This is the method we pass in a callLater() before we go to sleep.
+        self.sleeping = False
+        self.sleep_until = 0
+        self.recall = reactor.callLater(CLEAN_DUMP_SCHED, self.cleanup_and_dump_state)
+        log.msg(
+            eventid='cowrie.abuseipdb.wakeup',
+            format='AbuseIPDB plugin resuming activity after receiving '
+                   'Retry-After header in previous response.',
+        )
+
+    def clean_expired_timestamps(self, ip_key, current_time):
+        # Performs popleft() if leftmost timestamp has expired. Continues doing
+        # so until either; 1) a timestamp within our reporting window is
+        # reached, or; 2) the list is empty.
+        while self[ip_key]:
+            if not self[ip_key][0]:
+                break
+            elif self[ip_key][0] < current_time - self.tolerance_window:
+                self[ip_key].popleft()
+            else:
+                break
+
+    def find_and_delete_empty_entries(self):
+        # Search and destroy method. Iterates over dict, appends k to delete_me
+        # where v is an empty list.
+        delete_me = []
+        for k in self:
+            if not self[k]:
+                delete_me.append(k)
+        self.delete_entries(delete_me)
+
+    def delete_entries(self, delete_me):
+        for i in delete_me:
+            del self[i]
+
+    def can_rereport(self, ip_key, current_time):
+        # Checks if an IP in the logbook that has already been reported is
+        # ready to be re-reported again.
+        try:
+            if current_time > self[ip_key][1] + self.rereport_after:
+                return True
+
+            elif self[ip_key][0] and self.tolerance_attempts <= 1:
+                # If we were previously running with a tolerance_attempts > 1
+                # and have been been restarted with tolerance_attempts <= 1,
+                # we could still be carrying some logs which would evaluate as
+                # false in our first test. Reported IPs will still evaluate
+                # false here.
+                return True
+
+            else:
+                return False
+
+        except IndexError:
+            return True
+
+    def cleanup_and_dump_state(self, mode=0):
+        # Runs a full clean-up of logbook. Re-calls itself in CLEAN_DUMP_SCHED
+        # seconds. MODES: 0) Normal looping task, and; 1) Sleep/Stop mode;
+        # cancels any scheduled callLater() and doesn't recall itself.
+        if mode == 1:
+            try:
+                self.recall.cancel()
+            except AttributeError:
+                pass
+
+        if self.sleeping:
+            t = self.sleep_until
+        else:
+            t = time()
+
+        delete_me = []
+        for k in self:
+            if self.can_rereport(k, t):
+                delete_me.append(k)
+            self.clean_expired_timestamps(k, t)
+        self.delete_entries(delete_me)
+
+        self.find_and_delete_empty_entries()
+
+        self.dump_state()
+
+        if mode == 0 and not self.sleeping:
+            self.recall = reactor.callLater(CLEAN_DUMP_SCHED, self.cleanup_and_dump_state)
+
+    def dump_state(self):
+        dump = {
+            'sleeping': self.sleeping,
+            'sleep_until': self.sleep_until,
+            # Store current self_tolerance for comparison on next start
+            'tolerated': self.tolerance_attempts,
+        }
+
+        for k, v in self.items():
+            dump[k] = v
+
+        reactor.callInThread(self.write_dump_file, dump)
+
+    def write_dump_file(self, dump):
+        # Check self._writing; waits for release; timeout after 10 seconds.
+        i = 0
+        while self._writing:
+            sleep(1)
+            i += 1
+            if i >= 10:
+                return
+
+        # Acquire 'lock'
+        self._writing = True
+
+        with open(self.state_dump, 'wb') as f:
+            pickle.dump(dump, f, protocol=pickle.HIGHEST_PROTOCOL)
+
+        # Release 'lock'
+        self._writing = False
+
+
+class Reporter:
+    """ HTTP client and methods for preparing report paramaters.
+    """
+    def __init__(self, logbook, attempts):
+        self.logbook = logbook
+        self.attempts = attempts
+        self.headers = {
+            'User-Agent': 'Cowrie Honeypot AbuseIPDB plugin',
+            'Accept': 'application/json',
+            'Key': CowrieConfig().get('output_abuseipdb', 'api_key')
+        }
+
+    def report_ip_single(self, ip, t, uname):
+        self.logbook[ip] = (None, t)
+
+        t = self.epoch_to_string_utc(t)
+
+        params = {
+            'ip': ip,
+            'categories': '18,22',
+            'comment': 'Cowrie Honeypot: Unauthorised SSH/Telnet login attempt '
+                       'with user "{}" at {}'.format(uname, t)
+        }
+
+        self.http_request(params)
+
+    def report_ip_multiple(self, ip):
+        t_last = self.logbook[ip].pop()
+        t_first = self.epoch_to_string_utc(self.logbook[ip].popleft())
+
+        self.logbook[ip] = (None, t_last)
+
+        t_last = self.epoch_to_string_utc(t_last)
+
+        params = {
+            'ip': ip,
+            'categories': '18,22',
+            'comment': 'Cowrie Honeypot: {} unauthorised SSH/Telnet login attempts '
+                       'between {} and {}'.format(self.attempts, t_first, t_last)
+        }
+
+        self.http_request(params)
+
+    @staticmethod
+    def epoch_to_string_utc(t):
+        t_utc = datetime.utcfromtimestamp(t)
+        return t_utc.strftime('%Y-%m-%dT%H:%M:%SZ')
+
+    @staticmethod
+    def log_response_failed(ip, response, reason):
+        log.msg(
+            eventid='cowrie.abuseipdb.reportfail',
+            format='AbuseIPDB plugin failed to report IP %(IP)s. Received HTTP '
+                   'status code %(response)s in response. Reason: %(reason)s.',
+            IP=ip,
+            response=response,
+            reason=reason,
+        )
+
+    @defer.inlineCallbacks
+    def http_request(self, params):
+        try:
+            response = yield post(
+                url=ABUSEIP_URL,
+                headers=self.headers,
+                params=params,
+                )
+
+        except Exception as e:
+            log.msg(
+                eventid='cowrie.abuseipdb.reportfail',
+                format='AbuseIPDB plugin failed to report IP %(IP)s. '
+                       'Exception raised: %(exception)s.',
+                IP=params['ip'],
+                exception=repr(e),
+            )
+            return
+
+        if response.code != http.OK:
+            if response.code == 429:
+                return self.rate_limit_handler(params, response)
+
+            try:
+                reason = http.RESPONSES[response.code].decode('utf-8')
+            except Exception:
+                reason = 'Unable to determine.'
+
+            self.log_response_failed(params['ip'], response.code, reason)
+
+            return
+
+        j = yield response.json()
+
+        log.msg(
+            eventid='cowrie.abuseipdb.reportedip',
+            format='AbuseIPDB plugin successfully reported %(IP)s. Current '
+                   'AbuseIPDB confidence score for this IP is %(confidence)s',
+            IP=params['ip'],
+            confidence=j['data']['abuseConfidenceScore']
+        )
+
+    @defer.inlineCallbacks
+    def rate_limit_handler(self, params, response):
+        try:
+            j = yield response.json()
+            reason = j['errors'][0]['detail']
+
+        except (KeyError, JSONDecodeError):
+            reason = 'No other information provided or unexpected response'
+
+        self.log_response_failed(params['ip'], response.code, reason)
+
+        # AbuseIPDB will respond with a 429 and a Retry-After in its response
+        # headers if we've exceeded our limits for the day. Here we test for
+        # that header and, if it exists, put ourselves to sleep.
+        retry_after = yield response.headers.hasHeader('Retry-After')
+
+        if retry_after:
+            retry = yield response.headers.getRawHeaders('Retry-After')
+            retry = int(retry.pop())
+
+            if retry > 86340:
+                yield threads.deferToThread(self.sleeper_thread)
+
+                log.msg(
+                    eventid='cowrie.abuseipdb.ratelimited',
+                    format='AbuseIPDB plugin received Retry-After header > 86340 '
+                           'seconds in previous response. Possible delayed quota '
+                           'reset on AbuseIPDB servers; retrying request now.',
+                )
+
+                return self.http_request(params)
+
+            self.logbook.sleeping = True
+            self.logbook.sleep_until = time() + retry
+            reactor.callLater(retry, self.logbook.wakeup)
+            # It's not serious if we don't, but it's best to call the clean-up
+            # after logbook.sleeping has been set to True. The clean-up method
+            # checks for this flag and will use the wake-up time rather than
+            # the current time when sleep is set. mode=1 ensures we'll cancel
+            # any already scheduled calls to clean-up and don't schedule
+            # another one until the wake-up method calls it again.
+            self.logbook.cleanup_and_dump_state(mode=1)
+
+            self.epoch_to_string_utc(self.logbook.sleep_until)
+            log.msg(
+                eventid='cowrie.abuseipdb.ratelimited',
+                format='AbuseIPDB plugin received Retry-After header in '
+                       'response. Reporting activity will resume in '
+                       '%(retry_after)s seconds at %(wake_at)s',
+                retry_after=retry,
+                wake_at=self.epoch_to_string_utc(self.logbook.sleep_until)
+            )
+
+    def sleeper_thread(self):
+        # Cheap retry wait hack. Call in thread so as not to block.
+        sleep(10)


Test patch: