Repository: cowrie/cowrie

Issue:
Too many open files (EMFILE)
We're using Cowrie with a proxy-backend configuration. It's seeing quite a bit of brute forcing attempts, and something is causing the backend to leak file descriptors, eventually hitting Too many open files.

The symptoms are very similar to https://github.com/cowrie/cowrie/issues/367, although in our case, the attackers are still stuck at the brute forcing phase, so they haven't reached the honeypot contents yet.

From the backend's cowrie.log:
```
2020-06-08T14:16:22.505410Z [backend_pool.nat.ServerFactory#info] Starting factory <backend_pool.nat.ServerFactory object at 0x7fc035e2ce80>
2020-06-08T14:16:22.507706Z [backend_pool.nat.ClientFactory#info] Starting factory <backend_pool.nat.ClientFactory object at 0x7fc035e2c3c8>
2020-06-08T14:16:22.508111Z [twisted.internet.tcp.Port#info] EMFILE encountered; releasing reserved file descriptor.
2020-06-08T14:16:22.508193Z [twisted.internet.tcp.Port#info] Re-reserving EMFILE recovery file descriptor.
2020-06-08T14:16:22.508593Z [backend_pool.nat.ClientFactory#info] Stopping factory <backend_pool.nat.ClientFactory object at 0x7fc035e2c3c8>
2020-06-08T14:16:23.235361Z [twisted.internet.tcp.Port#info] EMFILE encountered; releasing reserved file descriptor.
2020-06-08T14:16:23.235508Z [twisted.internet.tcp.Port#info] EMFILE recovery: Closed socket from ('172.16.1.6', 42942)
2020-06-08T14:16:23.235537Z [twisted.internet.tcp.Port#info] Re-reserving EMFILE recovery file descriptor.
2020-06-08T14:16:24.463433Z [-] Unhandled Error
        Traceback (most recent call last):
          File "/srv/cowrie/cowrie-venv/lib/python3.6/site-packages/twisted/application/app.py", line 399, in startReactor

          File "/srv/cowrie/cowrie-venv/lib/python3.6/site-packages/twisted/application/app.py", line 312, in runReactorWithLogging

          File "/srv/cowrie/cowrie-venv/lib/python3.6/site-packages/twisted/internet/base.py", line 1283, in run

          File "/srv/cowrie/cowrie-venv/lib/python3.6/site-packages/twisted/internet/base.py", line 1292, in mainLoop

        --- <exception caught here> ---
          File "/srv/cowrie/cowrie-venv/lib/python3.6/site-packages/twisted/internet/base.py", line 913, in runUntilCurrent

          File "/srv/cowrie/cowrie-github/src/backend_pool/pool_service.py", line 280, in producer_loop

          File "/srv/cowrie/cowrie-github/src/backend_pool/pool_service.py", line 190, in __producer_check_health

          File "/srv/cowrie/cowrie-github/src/backend_pool/pool_service.py", line 151, in has_connectivity

          File "/srv/cowrie/cowrie-github/src/backend_pool/util.py", line 18, in nmap_port

          File "/usr/lib/python3.6/subprocess.py", line 423, in run

          File "/usr/lib/python3.6/subprocess.py", line 687, in __init__

          File "/usr/lib/python3.6/subprocess.py", line 1197, in _get_handles

        builtins.OSError: [Errno 24] Too many open files
```

Under /proc/*/fd, I can see the fd:s piling up.

```
[...]
lrwx------ 1 cowrie cowrie 64 Jun  9 06:30 85 -> 'socket:[127152879]'
lrwx------ 1 cowrie cowrie 64 Jun  9 06:30 86 -> 'socket:[127156659]'
lrwx------ 1 cowrie cowrie 64 Jun  9 06:30 88 -> 'socket:[127158533]'
lrwx------ 1 cowrie cowrie 64 Jun  9 06:30 89 -> 'socket:[127158534]'
[...]
```

Netstat also confirms the same.

```
[...]
twistd  586 cowrie  184u     IPv4          127199423      0t0       TCP *:42039 (LISTEN)
twistd  586 cowrie  185u     IPv4          127212415      0t0       TCP *:37159 (LISTEN)
twistd  586 cowrie  186u     IPv4          127217586      0t0       TCP *:41445 (LISTEN)
twistd  586 cowrie  187u     IPv4          127205735      0t0       TCP *:35751 (LISTEN)
[...]
```

On the proxy, the only clue I've found so far are these "Lost connection" entries:

`2020-06-09T06:36:34.074618Z [BackendSSHTransport,client] Lost connection with the pool backend: id 1`

However, looking at src/cowrie/ssh_proxy/client_transport.py lines 97 onwards, this "error" seems to be emitted by a rather normal client disconnect, and shouldn't be harmful?

https://github.com/cowrie/cowrie/blob/master/src/cowrie/ssh_proxy/client_transport.py#L97

Proxy config:

```
[honeypot]
backend = proxy
interactive_timeout = 180

[proxy]
backend = pool
pool_max_vms = 2
pool = remote
pool_host = 172.16.1.190
pool_port = 6415
backend_user = <redacted>
backend_pass = <redacted>

[ssh]
version = <redacted>
forwarding = false
```

Backend config:


```
[backend_pool]
pool_only = true
recycle_period = 3600
listen_endpoints = tcp:6415:interface=172.16.1.190
# disable telnet on the guests
guest_telnet_port = 0
guest_image_path = /srv/cowrie/minimal-4-ubuntu-18.04.qcow2
guest_hypervisor = qemu
guest_memory = 512
nat_public_ip = 172.16.1.190
```

I've also tried changing the interactive_timeout at the proxy, but it doesn't seem to affect the fd leakage.




Discussion:

Resolving patch:
diff --git a/src/backend_pool/nat.py b/src/backend_pool/nat.py
--- a/src/backend_pool/nat.py
+++ b/src/backend_pool/nat.py
@@ -74,16 +74,15 @@ def request_binding(self, guest_id, dst_ip, ssh_port, telnet_port):
         self.lock.acquire()
         try:
             # see if binding is already created
-            if dst_ip in self.bindings:
+            if guest_id in self.bindings:
                 # increase connected
                 self.bindings[guest_id][0] += 1
 
                 return self.bindings[guest_id][1]._realPortNumber, self.bindings[guest_id][2]._realPortNumber
-
             else:
                 nat_ssh = reactor.listenTCP(0, ServerFactory(dst_ip, ssh_port), interface='0.0.0.0')
                 nat_telnet = reactor.listenTCP(0, ServerFactory(dst_ip, telnet_port), interface='0.0.0.0')
-                self.bindings[guest_id] = [0, nat_ssh, nat_telnet]
+                self.bindings[guest_id] = [1, nat_ssh, nat_telnet]
 
                 return nat_ssh._realPortNumber, nat_telnet._realPortNumber
         finally:
@@ -94,10 +93,20 @@ def free_binding(self, guest_id):
         try:
             self.bindings[guest_id][0] -= 1
 
-            # stop listening if no-one connected
-            if self.bindings[guest_id][0] == 0:
+            # stop listening if no one is connected
+            if self.bindings[guest_id][0] <= 0:
                 self.bindings[guest_id][1].stopListening()
                 self.bindings[guest_id][2].stopListening()
+                del self.bindings[guest_id]
+        finally:
+            self.lock.release()
+
+    def free_all(self):
+        self.lock.acquire()
+        try:
 
+            for guest_id in self.bindings:
+                self.bindings[guest_id][1].stopListening()
+                self.bindings[guest_id][2].stopListening()
         finally:
             self.lock.release()
diff --git a/src/backend_pool/pool_server.py b/src/backend_pool/pool_server.py
--- a/src/backend_pool/pool_server.py
+++ b/src/backend_pool/pool_server.py
@@ -129,7 +129,7 @@ def __init__(self):
 
     def startFactory(self):
         # start the pool thread with default configs
-        self.pool_service = PoolService()
+        self.pool_service = PoolService(self.nat)
         self.pool_service.start_pool()
 
     def stopFactory(self):
diff --git a/src/backend_pool/pool_service.py b/src/backend_pool/pool_service.py
--- a/src/backend_pool/pool_service.py
+++ b/src/backend_pool/pool_service.py
@@ -33,8 +33,10 @@ class PoolService:
     A lock is required to manipulate VMs in states [available, using, used], since these are the ones that can be
     accessed by several consumers and the producer. All other states are accessed only by the single producer.
     """
-    def __init__(self):
+    def __init__(self, nat_service):
         self.qemu = backend_pool.libvirt.backend_service.LibvirtBackendService()
+        self.nat_service = nat_service
+
         self.guests = []
         self.guest_id = 0
         self.guest_lock = Lock()
@@ -52,6 +54,10 @@ def __init__(self):
         self.ssh_port = CowrieConfig().getint('backend_pool', 'guest_ssh_port', fallback=-1)
         self.telnet_port = CowrieConfig().getint('backend_pool', 'guest_telnet_port', fallback=-1)
 
+        self.local_pool = CowrieConfig().get('proxy', 'pool', fallback='local') == 'local'
+        self.pool_only = CowrieConfig().getboolean('backend_pool', 'pool_only', fallback=False)
+        self.use_nat = CowrieConfig().getboolean('backend_pool', 'use_nat', fallback=True)
+
         # detect invalid config
         if not self.ssh_port > 0 and not self.telnet_port > 0:
             log.msg(eventid='cowrie.backend_pool.service',
@@ -99,6 +105,11 @@ def stop_pool(self):
         # force destroy remaining stuff
         self.qemu.destroy_all_cowrie()
 
+        # close any NAT sockets
+        if not self.local_pool and self.use_nat or self.pool_only:
+            log.msg(eventid='cowrie.backend_pool.service', format='Free all NAT bindings')
+            self.nat_service.free_all()
+
         try:
             self.qemu.stop_backend()
         except libvirt.libvirtError:


Test patch: