Repository: cowrie/cowrie

Issue:
Overlapping in json logging
Hello,

Yesterday, I've updated to the newest version of cowrie and found that json logs regulary become invalid because of overlapping messages:

{"eventid": "cowrie.direct-tcpip.data", "timestamp": "2016-08-25T19:45:14.966466Z", "sensor": "redacted", "system": "SSHChannel None (13) on SSHService ssh-connection on HoneyPotSSHTransport,65,193.169.53.173", "isError": 0, "src_ip": "redacted", "session": "6649575c", "dst_port": 443, "dst_ip": "redacted", "data": "'\x16\x03\x01\x01x\x01\x00\x01t\x03\x03W\xbfKf\x82v^\x9e\x7f\x15\t\x08+o\x13(9\x05\xcd4\"\xebk}\xcb\xa00|\xce\x9e\x1do\x00\x00\xca\xc00\xc0,\xc0(\xc0$\xc0\x14\xc0\n\xc0\"\xc0!\x00\xa3\x00\x9f\x00k\x00j\x009\x008\x00\x88\x00\x87\xc0\x19\xc0 \x00\xa7\x00m\x00:\x00\x89\xc02\xc0.\xc0_\xc0&\xc0\x0f\xc0\x05\x00\x9d\x00=\x005\x00\x84\xc0\x12\xc0\x08\xc0\x1c\xc0\x1b\x00\x16\x00\x13\xc0\x17\xc0\x1a\x00\x1b\xc0\r\xc0\x03\x00\n\xc0/\xc0+\xc0\'\xc0#\xc0\x13\xc0\t\xc0\x1f\xc0\x1e\x00\xa2\x00\x9e\x00g\x00@\x003\x002\x00\x9a\x00\x99\x00E\x00D\xc0\x18\xc0\x1d\x00\xa6\x00l\x004\x00\x9b\x00F\xc01\xc0-\xc0)\xc0%\xc0\x0e\xc0\x04\x00\x9c\x00<\x00/\x00\x96\x00A\x00\x07\xc0\x11\xc0\x07\xc0\x16\x00\x18\xc0\x0c\xc0\x02\x00\x05\x00\x04\x00\x15\x00\x12\x00\x1a\x00\t\x00\x14\x00\x11\x00\x19\x00\x08\x00\x06\x00\x17\x00\x03\x00\xff\x01\x00\x00\x81\x00\x00\x00\x12\x00\x10\x00\x00\rwww.google.se\x00\x0b\x00\x04\x03\x00\x01\x02\x00\n\x004\x002\x00\x0e\x00\r\x00\x19\x00\x0b\x00\x0c\x00\x18\x00\t\x00\n\x00\x16\x00\x17\x00\x08\x00\x06\x00\x07\x00\x14\x00\x15\x00\x04\x00\x05\x00\x12\x00\x13\x00\x01\x00\x02\x00\x03\x00\x0f\x00\x10\x00\x11\x00\r\x00\"\x00 \x06\x01\x06\x02\x06\x03\x05\x01\x05\x02\x05\x03\x04\x01\x04\x02\x04\x03\x03\x01\x03\x02\x03\x03\x02\x01\x02\x02\x02\x03\x01\x01\x00\x0f\x00\x01\x01'", "message": "direct-tcp forward to 173.194.222.94:443 with data '\x16\x03\x01\x01x\x01\x00\x01t\x03\x03W\xbfKf\x82v^\x9e\x7f\x15\t\x08+o\x13(9\x05\xcd4\"\xebk}\xcb\xa00|\xce\x9e\x1do\x00\x00\xca\xc00\xc0,\xc0(\xc0$\xc0\x14\xc0\n\xc0\"\xc0!\x00\xa3\x00\x9f\x00k\x00j\x009\x008\x00\x88\x00\x87\xc0\x19\xc0 \x00\xa7\x00m\x00:\x00\x89\xc02\xc0.\xc0_\xc0&\xc0\x0f\xc0\x05\x00\x9d\x00=\x005\x00\x84\xc0\x12\xc0\x08\xc0\x1c\xc0\x1b\x00\x16\x00\x13\xc0\x17\xc0\x1a\x00\x1b\xc0\r\xc0\x03\x00\n\xc0/\xc0+\xc0\'\xc0#\xc0\x13\xc0\t\xc0\x1f\xc0\x1e\x00\xa2\x00\x9e\x00g\x00@\x003\x002\x00\x9a\x00\x99\x00E\x00D\xc0\x18\xc0\x1d\x00\xa6\x00l\x004\x00\x9b\x00F\xc01\xc0-\xc0)\xc0%\xc0\x0e\xc0\x04\x00\x9c\x00<\x00/\x00\x96\x00A\x00\x07\xc0\x11\xc0\x07\xc0\x16\x00\x18\xc0\x0c\xc0\x02\x00\x05\x00\x04\x00\x15\x00\x12\x00\x1a\x00\t\x00\x14\x00\x11\x00\x19\x00\x08\x00\x06\x00\x17\x00\x03\x00\xff\x01\x00\x00\x81\x00\x00\x00\x12\x00\x10\x00\x00\rwww.google.se\x00\x0b\x00\x04\x03\x00\x01\x02\x00\n\x004\x002\x00\x0e\x00\r\x00\x19\x00\x0b\x00\x0c\x00\x18\x00\t\x00\n\x00\x16\x00\x17\x00\x08\x00\x06\x00\x07\x00\x14\x00\x15\x00\x04\x00\x05\x00\x12\x00\x13\x00\x01\x00\x02\x00\x03\x00\x0f\x00\x10\x00\x11\x00\r{"eventid": "cowrie.direct-tcpip.data", "timestamp": "2016-08-25T19:45:14.966466Z", "sensor": "redacted", "system": "SSHChannel None (13) on SSHService ssh-connection on HoneyPotSSHTransport,65,193.169.53.1{"eventid": "cowrie.direct-tcpip.request", "timestamp": "2016-08-25T19:45:24.508664Z", "session": "a0c671d3", "src_port": 5556, "message": "direct-tcp connection request to 98.138.112.32:25 from localhost:5556", "system": "SSHService ssh-connection on HoneyPotSSHTransport,62,193.169.52.222", "isError": 0, "src_ip": "redacted", "dst_port": 25, "dst_ip": "redacted", "sensor": "redacted"} 

If you look carefully, there is the second log message inserted right into the first ( "\x11\x00\r{"eventid":" ) and the third is inserted into the second one: " 65,193.169.53.1{"eventid":  "

The only suspicious thing in logs is presence of tracebacks that repeat pretty often:

2016-08-25T22:29:08+0300 [SSHChannel None (2) on SSHService ssh-connection on HoneyPotSSHTransport,65,censored] sending close 2
2016-08-25T22:29:24+0300 [twisted.internet.defer#critical] Unhandled error in Deferred:

Traceback (most recent call last):
Failure: twisted.internet.error.ConnectionDone: Connection was closed cleanly.
...

2016-08-25T22:28:52+0300 [StripCrTelnetTransport,221,censored] Warning: state changed and new state returned
2016-08-25T22:29:01+0300 [twisted.internet.defer#critical] Unhandled error in Deferred:

Traceback (most recent call last):
Failure: twisted.conch.telnet.OptionRefused: twisted.conch.telnet.OptionRefused:'\x03'

Any ideas?



Discussion:
Interesting. Seems at high logging volume it will need some queuing system...

On Friday, 26 August 2016, fe7ch notifications@github.com wrote:

> Hello,
> 
> Yesterday, I've updated to the newest version of cowrie and found that
> json logs regulary become invalid because of overlapping messages:
> 
> {"eventid": "cowrie.direct-tcpip.data", "timestamp":
> "2016-08-25T19:45:14.966466Z", "sensor": "redacted", "system": "SSHChannel
> None (13) on SSHService ssh-connection on HoneyPotSSHTransport,65,193.169.53.173",
> "isError": 0, "src_ip": "redacted", "session": "6649575c", "dst_port": 443,
> "dst_ip": "redacted", "data": "'\x16\x03\x01\x01x\x01\x00\
> x01t\x03\x03W\xbfKf\x82v^\x9e\x7f\x15\t\x08+o\x13(9\x05\
> xcd4\"\xebk}\xcb\xa00|\xce\x9e\x1do\x00\x00\xca\xc00\xc0,
> \xc0(\xc0$\xc0\x14\xc0\n\xc0\"\xc0!\x00\xa3\x00\x9f\x00k\
> x00j\x009\x008\x00\x88\x00\x87\xc0\x19\xc0 \x00\xa7\x00m\x00:\x00\x89\
> xc02\xc0.\xc0_\xc0&\xc0\x0f\xc0\x05\x00\x9d\x00=\x005\x00\x84\xc0\x12\xc0\x08\xc0\x1c\xc0\x1b\x00\x16\x00\x13\xc0\x17\xc0\x1a\x00\x1b\xc0\r\xc0\x03\x00\n\xc0/\xc0+\xc0\'\xc0#\xc0\x13\xc0\t\xc0\x1f\xc0\x1e\x00\xa2\x00\x9e\x00g\x00@\x003\x002\x00\x9a\x00\x99\x00E\x00D\xc0\x18\xc0\x1d\x00\xa6\x00l\x004\x00\x9b\x00F\xc01\xc0-\xc0)\xc0%\xc0\x0e\xc0\x04\x00\x9c\x00<\x00/\x00\x96\x00A\x00\x07\xc0\x11\xc0\x07\xc0\x16\x00\x18\xc0\x0c\xc0\x02\x00\x05\x00\x04\x00\x15\x00\x12\x00\x1a\x00\t\x00\x14\x00\x11\x00\x19\x00\x08\x00\x06\x00\x17\x00\x03\x00\xff\x01\x00\x00\x81\x00\x00\x00\x12\x00\x10\x00\x00\rwww.google.se
> http://rwww.google.se\x00\x0b\x00\x04\x03\x00\x01\x02\x00\n\x004\x002\x00\x0e\x00\r\x00\x19\x00\x0b\x00\x0c\x00\x18\x00\t\x00\n\x00\x16\x00\x17\x00\x08\x00\x06\x00\x07\x00\x14\x00\x15\x00\x04\x00\x05\x00\x12\x00\x13\x00\x01\x00\x02\x00\x03\x00\x0f\x00\x10\x00\x11\x00\r\x00\"\x00
> \x06\x01\x06\x02\x06\x03\x05\x01\x05\x02\x05\x03\x04\x01\x04\x02\x04\x03\x03\x01\x03\x02\x03\x03\x02\x01\x02\x02\x02\x03\x01\x01\x00\x0f\x00\x01\x01'",
> "message": "direct-tcp forward to 173.194.222.94:443
> http://173.194.222.94:443 with data
> '\x16\x03\x01\x01x\x01\x00\x01t\x03\x03W\xbfKf\x82v^\x9e\x7f\x15\t\x08+o\x13(9\x05\xcd4\"\xebk}\xcb\xa00|\xce\x9e\x1do\x00\x00\xca\xc00\xc0,\xc0(\xc0$\xc0\x14\xc0\n\xc0\"\xc0!\x00\xa3\x00\x9f\x00k\x00j\x009\x008\x00\x88\x00\x87\xc0\x19\xc0
> \x00\xa7\x00m\x00:\x00\x89\xc02\xc0.\xc0_\xc0&\xc0\x0f\
> xc0\x05\x00\x9d\x00=\x005\x00\x84\xc0\x12\xc0\x08\xc0\x1c\
> xc0\x1b\x00\x16\x00\x13\xc0\x17\xc0\x1a\x00\x1b\xc0\r\xc0\
> x03\x00\n\xc0/\xc0+\xc0\'\xc0#\xc0\x13\xc0\t\xc0\x1f\xc0\
> x1e\x00\xa2\x00\x9e\x00g\x00@\x003\x002\x00\x9a\x00\x99\
> x00E\x00D\xc0\x18\xc0\x1d\x00\xa6\x00l\x004\x00\x9b\x00F\
> xc01\xc0-\xc0)\xc0%\xc0\x0e\xc0\x04\x00\x9c\x00<\x00/\x00\
> x96\x00A\x00\x07\xc0\x11\xc0\x07\xc0\x16\x00\x18\xc0\x0c\
> xc0\x02\x00\x05\x00\x04\x00\x15\x00\x12\x00\x1a\x00\t\x00\
> x14\x00\x11\x00\x19\x00\x08\x00\x06\x00\x17\x00\x03\x00\
> xff\x01\x00\x00\x81\x00\x00\x00\x12\x00\x10\x00\x00\rwww.google.se
> \x00\x0b\x00\x04\x03\x00\x01\x02\x00\n\x004\x002\
> x00\x0e\x00\r\x00\x19\x00\x0b\x00\x0c\x00\x18\x00\t\x00\n\
> x00\x16\x00\x17\x00\x08\x00\x06\x00\x07\x00\x14\x00\x15\
> x00\x04\x00\x05\x00\x12\x00\x13\x00\x01\x00\x02\x00\x03\
> x00\x0f\x00\x10\x00\x11\x00\r{"eventid": "cowrie.direct-tcpip.data",
> "timestamp": "2016-08-25T19:45:14.966466Z", "sensor": "redacted", "system":
> "SSHChannel None (13) on SSHService ssh-connection on
> HoneyPotSSHTransport,65,193.169.53.1{"eventid":
> "cowrie.direct-tcpip.request", "timestamp": "2016-08-25T19:45:24.508664Z",
> "session": "a0c671d3", "src_port": 5556, "message": "direct-tcp connection
> request to 98.138.112.32:25 from localhost:5556", "system": "SSHService
> ssh-connection on HoneyPotSSHTransport,62,193.169.52.222", "isError": 0,
> "src_ip": "redacted", "dst_port": 25, "dst_ip": "redacted", "sensor":
> "redacted"}
> 
> If you look carefully, there is the second log message inserted right into
> the first ( "\x11\x00\r{"eventid":" ) and the third is inserted into the
> second one: " 65,193.169.53.1{"eventid": "
> 
> The only suspicious thing in logs is presence of tracebacks that repeat
> pretty often:
> 
> 2016-08-25T22:29:08+0300 [SSHChannel None (2) on SSHService ssh-connection
> on HoneyPotSSHTransport,65,censored] sending close 2
> 2016-08-25T22:29:24+0300 [twisted.internet.defer#critical] Unhandled
> error in Deferred:
> 
> Traceback (most recent call last):
> Failure: twisted.internet.error.ConnectionDone: Connection was closed
> cleanly.
> ...
> 
> 2016-08-25T22:28:52+0300 [StripCrTelnetTransport,221,censored] Warning:
> state changed and new state returned
> 2016-08-25T22:29:01+0300 [twisted.internet.defer#critical] Unhandled
> error in Deferred:
> 
> Traceback (most recent call last):
> Failure: twisted.conch.telnet.OptionRefused: twisted.conch.telnet.
> OptionRefused:'\x03'
> 
> Any ideas?
> 
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> https://github.com/micheloosterhof/cowrie/issues/230, or mute the thread
> https://github.com/notifications/unsubscribe-auth/ABA4g8z0PhkgXZTvb_IZQ8RDk5Hbdr6-ks5qjrrdgaJpZM4Jt7oF
> .

I've run into the same issue. I'm currently testing the queue approach @micheloosterhof mentioned, so PR coming soon.


Resolving patch:
diff --git a/cowrie/output/jsonlog.py b/cowrie/output/jsonlog.py
--- a/cowrie/output/jsonlog.py
+++ b/cowrie/output/jsonlog.py
@@ -32,7 +32,10 @@
 
 import json
 import os
+import Queue
+import threading
 
+from twisted.python import log
 import twisted.python.logfile
 
 import cowrie.core.output
@@ -47,8 +50,21 @@ def __init__(self, cfg):
         fn = cfg.get('output_jsonlog', 'logfile')
         dirs = os.path.dirname(fn)
         base = os.path.basename(fn)
+
+        # create the log queue with a default buffer size if none is specified in
+        # the log file.
+        buffer_size = 10000
+        if cfg.has_option('output_jsonlog', 'buffer_size'):
+            buffer_size = int(cfg.get('output_jsonlog', 'buffer_size'))
+        self._log_writer_queue = Queue.Queue(maxsize=buffer_size)
+
+        # allocate the output file
         self.outfile = twisted.python.logfile.DailyLogFile(base, dirs, defaultMode=0o664)
 
+        # start the log writer thread
+        self._log_writer_thread = threading.Thread(target=self._write_log)
+        self._log_writer_thread.daemon = True
+        self._log_writer_thread.start()
 
     def start(self):
         """
@@ -59,6 +75,7 @@ def start(self):
     def stop(self):
         """
         """
+        self._log_queue.join()
         self.outfile.flush()
 
 
@@ -69,7 +86,26 @@ def write(self, logentry):
             # Remove twisted 15 legacy keys
             if i.startswith('log_'):
                 del logentry[i]
-        json.dump(logentry, self.outfile)
-        self.outfile.write('\n')
-        self.outfile.flush()
+
+        # TODO: There's a possibility that the queue is full when we do this put, which means
+        # we'll lose the log item. We specifically use put_nowait so in that case it doesn't
+        # block the main writer thread.
+        try:
+            self._log_writer_queue.put_nowait(json.dumps(logentry))
+        except Queue.Full:
+            log.err('Could not queue jsonlog item. Consider increasing buffer_size in [output_jsonlog] of your cowrie configuration')
+
+    def _write_log(self):
+        # there's a probability of hitting IO errors while attempting to write
+        # for various reasons (for example, the disk is full). So, regardless
+        # of what happens during the write, we always mark the queue item as done
+        # so self.stop() can properly join on any remaining items.
+        while True:
+            item = self._log_writer_queue.get()
+            try:
+                self.outfile.write(item)
+                self.outfile.write('\n')
+                self.outfile.flush()
+            finally:
+                self._log_writer_queue.task_done()
 


Test patch: