Repository: cowrie/cowrie

Issue:
HPFeeds3 UnicodeDecodeError in ttylog.read().encode().hex()
**Describe the bug**
Stack Trace from the cowrie version v2.3.0, as already described in #1307 

```
cowrie           | 2022-01-23T14:52:17+0000 [twisted.logger._observer#critical] Temporarily disabling observer LegacyLogObserverWrapper(<bound method Output.emit of <cowrie.output.hpfeeds3.Output object at 0x7f4019656490>>) due to exception: [Failure instance: Traceback: <class 'UnicodeDecodeError'>: 'utf-8' codec can't decode byte 0x88 in position 16: invalid start byte
cowrie           | 	/home/cowrie/cowrie/src/cowrie/insults/insults.py:226:connectionLost
cowrie           | 	/usr/lib/python3.9/site-packages/twisted/python/threadable.py:51:sync
cowrie           | 	/usr/lib/python3.9/site-packages/twisted/python/log.py:281:msg
cowrie           | 	/usr/lib/python3.9/site-packages/twisted/logger/_legacy.py:147:publishToNewObserver
cowrie           | 	--- <exception caught here> ---
cowrie           | 	/usr/lib/python3.9/site-packages/twisted/logger/_observer.py:82:__call__
cowrie           | 	/usr/lib/python3.9/site-packages/twisted/logger/_legacy.py:90:__call__
cowrie           | 	/home/cowrie/cowrie/src/cowrie/core/output.py:240:emit
cowrie           | 	/home/cowrie/cowrie/src/cowrie/output/hpfeeds3.py:110:write
cowrie           | 	/usr/lib/python3.9/codecs.py:322:decode
cowrie           | 	]
cowrie           | 	Traceback (most recent call last):
cowrie           | 	  File "/home/cowrie/cowrie/src/cowrie/insults/insults.py", line 226, in connectionLost
cowrie           | 	    log.msg(
cowrie           | 	  File "/usr/lib/python3.9/site-packages/twisted/python/threadable.py", line 51, in sync
cowrie           | 	    return function(self, *args, **kwargs)
cowrie           | 	  File "/usr/lib/python3.9/site-packages/twisted/python/log.py", line 281, in msg
cowrie           | 	    _publishNew(self._publishPublisher, actualEventDict, textFromEventDict)
cowrie           | 	  File "/usr/lib/python3.9/site-packages/twisted/logger/_legacy.py", line 147, in publishToNewObserver
cowrie           | 	    observer(eventDict)
cowrie           | 	--- <exception caught here> ---
cowrie           | 	  File "/usr/lib/python3.9/site-packages/twisted/logger/_observer.py", line 82, in __call__
cowrie           | 	    observer(event)
cowrie           | 	  File "/usr/lib/python3.9/site-packages/twisted/logger/_legacy.py", line 90, in __call__
cowrie           | 	    self.legacyObserver(event)
cowrie           | 	  File "/home/cowrie/cowrie/src/cowrie/core/output.py", line 240, in emit
cowrie           | 	    self.write(ev)
cowrie           | 	  File "/home/cowrie/cowrie/src/cowrie/output/hpfeeds3.py", line 110, in write
cowrie           | 	    self.meta[session]["ttylog"] = ttylog.read().encode().hex()
cowrie           | 	  File "/usr/lib/python3.9/codecs.py", line 322, in decode
cowrie           | 	    (result, consumed) = self._buffer_decode(data, self.errors, final)
cowrie           | 	builtins.UnicodeDecodeError: 'utf-8' codec can't decode byte 0x88 in position 16: invalid start byte
```

**Server (please complete the following information):**
 - OS: Alpine Linux in Docker
 - Python: Python 3.9

**Additional context**
The ttylog seems to be a binary file with only parts of it being text. 

At the moment the file is opened as a text file, then encoded to utf-8 bytes and then to a hex representation. Opening it as a binary file and directly transforming it to a hex reprenstation should fix it.
HPFeeds3 UnicodeDecodeError in ttylog.read().encode().hex()
**Describe the bug**
Stack Trace from the cowrie version v2.3.0, as already described in #1307 

```
cowrie           | 2022-01-23T14:52:17+0000 [twisted.logger._observer#critical] Temporarily disabling observer LegacyLogObserverWrapper(<bound method Output.emit of <cowrie.output.hpfeeds3.Output object at 0x7f4019656490>>) due to exception: [Failure instance: Traceback: <class 'UnicodeDecodeError'>: 'utf-8' codec can't decode byte 0x88 in position 16: invalid start byte
cowrie           | 	/home/cowrie/cowrie/src/cowrie/insults/insults.py:226:connectionLost
cowrie           | 	/usr/lib/python3.9/site-packages/twisted/python/threadable.py:51:sync
cowrie           | 	/usr/lib/python3.9/site-packages/twisted/python/log.py:281:msg
cowrie           | 	/usr/lib/python3.9/site-packages/twisted/logger/_legacy.py:147:publishToNewObserver
cowrie           | 	--- <exception caught here> ---
cowrie           | 	/usr/lib/python3.9/site-packages/twisted/logger/_observer.py:82:__call__
cowrie           | 	/usr/lib/python3.9/site-packages/twisted/logger/_legacy.py:90:__call__
cowrie           | 	/home/cowrie/cowrie/src/cowrie/core/output.py:240:emit
cowrie           | 	/home/cowrie/cowrie/src/cowrie/output/hpfeeds3.py:110:write
cowrie           | 	/usr/lib/python3.9/codecs.py:322:decode
cowrie           | 	]
cowrie           | 	Traceback (most recent call last):
cowrie           | 	  File "/home/cowrie/cowrie/src/cowrie/insults/insults.py", line 226, in connectionLost
cowrie           | 	    log.msg(
cowrie           | 	  File "/usr/lib/python3.9/site-packages/twisted/python/threadable.py", line 51, in sync
cowrie           | 	    return function(self, *args, **kwargs)
cowrie           | 	  File "/usr/lib/python3.9/site-packages/twisted/python/log.py", line 281, in msg
cowrie           | 	    _publishNew(self._publishPublisher, actualEventDict, textFromEventDict)
cowrie           | 	  File "/usr/lib/python3.9/site-packages/twisted/logger/_legacy.py", line 147, in publishToNewObserver
cowrie           | 	    observer(eventDict)
cowrie           | 	--- <exception caught here> ---
cowrie           | 	  File "/usr/lib/python3.9/site-packages/twisted/logger/_observer.py", line 82, in __call__
cowrie           | 	    observer(event)
cowrie           | 	  File "/usr/lib/python3.9/site-packages/twisted/logger/_legacy.py", line 90, in __call__
cowrie           | 	    self.legacyObserver(event)
cowrie           | 	  File "/home/cowrie/cowrie/src/cowrie/core/output.py", line 240, in emit
cowrie           | 	    self.write(ev)
cowrie           | 	  File "/home/cowrie/cowrie/src/cowrie/output/hpfeeds3.py", line 110, in write
cowrie           | 	    self.meta[session]["ttylog"] = ttylog.read().encode().hex()
cowrie           | 	  File "/usr/lib/python3.9/codecs.py", line 322, in decode
cowrie           | 	    (result, consumed) = self._buffer_decode(data, self.errors, final)
cowrie           | 	builtins.UnicodeDecodeError: 'utf-8' codec can't decode byte 0x88 in position 16: invalid start byte
```

**Server (please complete the following information):**
 - OS: Alpine Linux in Docker
 - Python: Python 3.9

**Additional context**
The ttylog seems to be a binary file with only parts of it being text. 

At the moment the file is opened as a text file, then encoded to utf-8 bytes and then to a hex representation. Opening it as a binary file and directly transforming it to a hex reprenstation should fix it.


Discussion:

Resolving patch:
diff --git a/src/cowrie/output/hpfeeds3.py b/src/cowrie/output/hpfeeds3.py
--- a/src/cowrie/output/hpfeeds3.py
+++ b/src/cowrie/output/hpfeeds3.py
@@ -106,8 +106,8 @@ def write(self, entry):
 
         elif entry["eventid"] == "cowrie.log.closed":
             # entry["ttylog"]
-            with open(entry["ttylog"]) as ttylog:
-                self.meta[session]["ttylog"] = ttylog.read().encode().hex()
+            with open(entry["ttylog"], 'rb') as ttylog:
+                self.meta[session]["ttylog"] = ttylog.read().hex()
 
         elif entry["eventid"] == "cowrie.session.closed":
             meta = self.meta.pop(session, None)


Test patch: