Repository: learningequality/kolibri

Issue:
unhandled zeroconf exception on shutdown

### Observed behavior

when shutting down Kolibri, I see unhandled errors from zeroconf processes.

```
(k013x) d@slate ~/P/l/k013x (release-v0.13.x)> kolibri start
WARNING:root:No C Extensions available for this platform.

INFO     Option RUN_MODE in section [Deployment] being overridden by environment variable KOLIBRI_RUN_MODE
INFO     Running Kolibri with the following settings: kolibri.deployment.default.settings.base
INFO     Running Kolibri
INFO     Kolibri running on:

	http://172.27.5.138:8080/
	http://127.0.0.1:8080/

INFO     Starting Kolibri 0.13.0b7.dev0+git.116.g19a41325
INFO     ENGINE Listening for SIGHUP.
INFO     ENGINE Listening for SIGUSR1.
INFO     ENGINE Listening for SIGTERM.
INFO     ENGINE Listening for SIGINT.
INFO     ENGINE Bus STARTING
INFO     Started new JOBCHECKER thread ID#a095c6d41ff2434d8c88a09f205483f4
INFO     Started new JOBCHECKER thread ID#a6d339d9adfa4b398e3cd4c15afb9c70
INFO     Started new SCHEDULECHECKER thread ID#e0c08d99b51f4946ac02c008d6c0eddc
INFO     Registering ourselves to zeroconf network with id '4d7f0be7ea562102062ec84450a1d357'...
INFO     ENGINE Serving on http://0.0.0.0:8080
INFO     ENGINE Bus STARTED
INFO     Ping succeeded! (response: {u'messages': [], u'id': 513065})
^[[1;5INFO     127.0.0.1 - - "GET /static/kolibri.core.default_frontend/kolibri.core.default_frontend0.13.0b7.dev0+git.116.g19a41325.css HTTP/1.1" 404 489 "http://localhost:8080/en/facility/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.79 Safari/537.36"
INFO     127.0.0.1 - - "GET /static/assets/fonts/noto-subset.en.css?v=0.13.0b7.dev0+git.116.g19a41325 HTTP/1.1" 200 16039 "http://localhost:8080/en/facility/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.79 Safari/537.36"
INFO     127.0.0.1 - - "GET /static/kolibri.plugins.facility.app/kolibri.plugins.facility.app0.13.0b7.dev0+git.116.g19a41325.css HTTP/1.1" 404 488 "http://localhost:8080/en/facility/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.79 Safari/537.36"
INFO     127.0.0.1 - - "GET /static/kolibri.core.default_frontend/kolibri.core.default_frontend-0.13.0b7.dev0+git.116.g19a41325.js HTTP/1.1" 404 488 "http://localhost:8080/en/facility/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.79 Safari/537.36"
INFO     127.0.0.1 - - "GET /static/kolibri.plugins.facility.app/kolibri.plugins.facility.app-0.13.0b7.dev0+git.116.g19a41325.js HTTP/1.1" 404 489 "http://localhost:8080/en/facility/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.79 Safari/537.36"
INFO     127.0.0.1 - - "GET /static/assets/fonts/noto-common.css?v=0.13.0b7.dev0+git.116.g19a41325 HTTP/1.1" 200 108608 "http://localhost:8080/en/facility/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.79 Safari/537.36"
INFO     127.0.0.1 - - "GET /static/kolibri.core.default_frontend/kolibri.core.default_frontend0.13.0b7.dev0+git.116.g19a41325.css HTTP/1.1" 404 489 "http://localhost:8080/en/facility/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.79 Safari/537.36"
INFO     127.0.0.1 - - "GET /static/assets/fonts/noto-full.en.modern.css?v=0.13.0b7.dev0+git.116.g19a41325 HTTP/1.1" 200 2549 "http://localhost:8080/en/facility/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.79 Safari/537.36"
INFO     127.0.0.1 - - "GET /static/kolibri.core.default_frontend/kolibri.core.default_frontend-0.13.0b7.dev0+git.116.g19a41325.js HTTP/1.1" 404 488 "http://localhost:8080/en/facility/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.79 Safari/537.36"
INFO     127.0.0.1 - - "GET /static/kolibri.plugins.facility.app/kolibri.plugins.facility.app0.13.0b7.dev0+git.116.g19a41325.css HTTP/1.1" 404 488 "http://localhost:8080/en/facility/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.79 Safari/537.36"
INFO     127.0.0.1 - - "GET /static/kolibri.plugins.facility.app/kolibri.plugins.facility.app-0.13.0b7.dev0+git.116.g19a41325.js HTTP/1.1" 404 489 "http://localhost:8080/en/facility/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.79 Safari/537.36"
^CINFO     ENGINE Caught signal SIGINT.
INFO     ENGINE Bus STOPPING
^CINFO     ENGINE Caught signal SIGINT.
INFO     ENGINE Bus STOPPING
INFO     ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('0.0.0.0', 8080)) already shut down
WARNING  SCHEDULECHECKER shut down event received; closing.
INFO     
Kolibri instance '4d7f0be7ea562102062ec84450a1d357' has left the zeroconf network.

WARNING  JOBCHECKER shut down event received; closing.
WARNING  JOBCHECKER shut down event received; closing.
INFO     ENGINE Bus STOPPED
INFO     ENGINE Bus EXITING
INFO     ENGINE Bus EXITED
INFO     ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('0.0.0.0', 8080)) shut down
INFO     ENGINE Bus STOPPED
INFO     ENGINE Bus EXITING
INFO     ENGINE Bus EXITED
INFO     ENGINE Waiting for child threads to terminate...
Exception in thread expire_cache (most likely raised during interpreter shutdown):Exception in thread zeroconf-Reaper (most likely raised during interpreter shutdown):

Traceback (most recent call last):
  File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 801, in __bootstrap_inner
  File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 754, in run
  File "/Users/d/Projects/le/k013x/.venv/lib/python2.7/site-packages/cherrypy/lib/caching.py", line 199, in expire_cache
<type 'exceptions.TypeError'>: 'NoneType' object is not callable
Traceback (most recent call last):
  File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 801, in __bootstrap_inner
  File "/Users/d/Projects/le/k013x/.venv/lib/python2.7/site-packages/zeroconf.py", line 1264, in run
  File "/Users/d/Projects/le/k013x/.venv/lib/python2.7/site-packages/zeroconf.py", line 1838, in wait
  File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 289, in __exit__
  File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 216, in __exit__
  File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 203, in release
<type 'exceptions.TypeError'>: 'NoneType' object is not callable
```

### Expected behavior

no unhandled errors

### User-facing consequences

unknown


### Steps to reproduce

* Start Kolibri
* Shut it down



### Context

0.13.0 beta, after #6380 was merged



Discussion:

Resolving patch:
diff --git a/kolibri/core/discovery/utils/network/search.py b/kolibri/core/discovery/utils/network/search.py
--- a/kolibri/core/discovery/utils/network/search.py
+++ b/kolibri/core/discovery/utils/network/search.py
@@ -1,4 +1,3 @@
-import atexit
 import json
 import logging
 import socket
@@ -45,14 +44,14 @@ def __init__(self, id, port=8080, data={}):
         self.port = port
         self.data = {key: json.dumps(val) for (key, val) in data.items()}
 
-        atexit.register(self.cleanup)
-
     def register(self):
 
         if not ZEROCONF_STATE["zeroconf"]:
             initialize_zeroconf_listener()
 
-        assert self.info is None, "Service is already registered!"
+        if self.info is not None:
+            logger.error("Service is already registered!")
+            return
 
         i = 1
         id = self.id
@@ -88,7 +87,9 @@ def register(self):
 
     def unregister(self):
 
-        assert self.info is not None, "Service is not registered!"
+        if self.info is None:
+            logging.error("Service is not registered!")
+            return
 
         ZEROCONF_STATE["zeroconf"].unregister_service(self.info)
 
@@ -105,7 +106,15 @@ class KolibriZeroconfListener(object):
     instances = {}
 
     def add_service(self, zeroconf, type, name):
-        info = zeroconf.get_service_info(type, name)
+        timeout = 5000
+        info = zeroconf.get_service_info(type, name, timeout=timeout)
+        if info is None:
+            logger.warn(
+                "Zeroconf network service information could not be retrieved within {} seconds".format(
+                    str(timeout / 1000.0)
+                )
+            )
+            return
         id = _id_from_name(name)
         ip = socket.inet_ntoa(info.address)
 
@@ -140,7 +149,7 @@ def add_service(self, zeroconf, type, name):
                 )
 
                 logger.info(
-                    "Kolibri instance '%s' joined zeroconf network; service info: %s\n"
+                    "Kolibri instance '%s' joined zeroconf network; service info: %s"
                     % (id, self.instances[id])
                 )
 
@@ -148,11 +157,14 @@ def add_service(self, zeroconf, type, name):
                 import traceback
 
                 logger.warn(
-                    (
-                        "A new Kolibri instance '%s' was seen on the zeroconf network, "
-                        + "but we had trouble getting the information we needed about it.  This probably isn't a big deal!\n\n"
-                        + "service info:\n %s;\n\nThe following exception was raised:\n%s"
-                    )
+                    """
+                        A new Kolibri instance '%s' was seen on the zeroconf network,
+                        but we had trouble getting the information we needed about it.
+                        Service info:
+                        %s
+                        The following exception was raised:
+                        %s
+                        """
                     % (id, self.instances[id], traceback.format_exc(limit=1)),
                 )
             finally:
@@ -160,7 +172,7 @@ def add_service(self, zeroconf, type, name):
 
     def remove_service(self, zeroconf, type, name):
         id = _id_from_name(name)
-        logger.info("\nKolibri instance '%s' has left the zeroconf network.\n" % (id,))
+        logger.info("Kolibri instance '%s' has left the zeroconf network." % (id,))
 
         try:
             if id in self.instances:
@@ -192,6 +204,7 @@ def unregister_zeroconf_service():
     if ZEROCONF_STATE["service"] is not None:
         ZEROCONF_STATE["service"].cleanup()
     ZEROCONF_STATE["service"] = None
+    ZEROCONF_STATE["zeroconf"].close()
 
 
 def initialize_zeroconf_listener():


Test patch:
diff --git a/kolibri/core/discovery/test/test_network_search.py b/kolibri/core/discovery/test/test_network_search.py
--- a/kolibri/core/discovery/test/test_network_search.py
+++ b/kolibri/core/discovery/test/test_network_search.py
@@ -46,6 +46,7 @@ class MockZeroconf(Zeroconf):
     def __init__(self, *args, **kwargs):
         self.browsers = {}
         self.services = {}
+        self._GLOBAL_DONE = True
 
     def get_service_info(self, type_, name, timeout=3000):
         id = _id_from_name(name)