Repository: evennia/evennia

Issue:
[BUG] Django reports MySQL server has gone away (similar to #1376)
I have a fairly recent and fresh install of Evennia 0.9.5, and it appears that old DB connection issues are still around.  If the server sits around for awhile, it stops functioning - opening up the webclient greets the user with "The connection was closed or lost."

Evennia's portal.log shows:
```
21-03-19 18:22:59  |Portal| Unhandled Error
        Traceback (most recent call last):
          File "/home/evennia/muddev/evenv/lib/python3.7/site-packages/twisted/web/http.py", line 1755, in dataReceived
            finishCallback(data[contentLength:])
          File "/home/evennia/muddev/evenv/lib/python3.7/site-packages/twisted/web/http.py", line 2171, in _finishRequestBody
            self.allContentReceived()
          File "/home/evennia/muddev/evenv/lib/python3.7/site-packages/twisted/web/http.py", line 2284, in allContentReceived
            req.requestReceived(command, path, version)
          File "/home/evennia/muddev/evenv/lib/python3.7/site-packages/twisted/web/http.py", line 946, in requestReceived
            self.process()
        --- <exception caught here> ---
          File "/home/evennia/muddev/evenv/lib/python3.7/site-packages/twisted/web/server.py", line 235, in process
            self.render(resrc)
          File "/home/evennia/muddev/evenv/lib/python3.7/site-packages/twisted/web/server.py", line 302, in render
            body = resrc.render(self)
          File "/home/evennia/muddev/evenv/lib/python3.7/site-packages/twisted/web/resource.py", line 265, in render
            return m(request)
          File "/home/evennia/muddev/evennia/evennia/server/portal/webclient_ajax.py", line 313, in render_POST
            return self.mode_init(request)
          File "/home/evennia/muddev/evennia/evennia/server/portal/webclient_ajax.py", line 207, in mode_init
            uid = csession and csession.get("webclient_authenticated_uid", False)
          File "/home/evennia/muddev/evenv/lib/python3.7/site-packages/django/contrib/sessions/backends/base.py", line 65, in get
            return self._session.get(key, default)
          File "/home/evennia/muddev/evenv/lib/python3.7/site-packages/django/contrib/sessions/backends/base.py", line 194, in _get_session
            self._session_cache = self.load()
          File "/home/evennia/muddev/evenv/lib/python3.7/site-packages/django/contrib/sessions/backends/db.py", line 43, in load
            s = self._get_session_from_db()
          File "/home/evennia/muddev/evenv/lib/python3.7/site-packages/django/contrib/sessions/backends/db.py", line 34, in _get_session_from_db
            expire_date__gt=timezone.now()
          File "/home/evennia/muddev/evenv/lib/python3.7/site-packages/django/db/models/manager.py", line 82, in manager_method
            return getattr(self.get_queryset(), name)(*args, **kwargs)
          File "/home/evennia/muddev/evenv/lib/python3.7/site-packages/django/db/models/query.py", line 402, in get
            num = len(clone)
          File "/home/evennia/muddev/evenv/lib/python3.7/site-packages/django/db/models/query.py", line 256, in __len__
            self._fetch_all()
          File "/home/evennia/muddev/evenv/lib/python3.7/site-packages/django/db/models/query.py", line 1242, in _fetch_all
            self._result_cache = list(self._iterable_class(self))
          File "/home/evennia/muddev/evenv/lib/python3.7/site-packages/django/db/models/query.py", line 55, in __iter__
            results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
          File "/home/evennia/muddev/evenv/lib/python3.7/site-packages/django/db/models/sql/compiler.py", line 1142, in execute_sql
            cursor.execute(sql, params)
          File "/home/evennia/muddev/evenv/lib/python3.7/site-packages/django/db/backends/utils.py", line 67, in execute
            return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
          File "/home/evennia/muddev/evenv/lib/python3.7/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
            return executor(sql, params, many, context)
          File "/home/evennia/muddev/evenv/lib/python3.7/site-packages/django/db/backends/utils.py", line 84, in _execute
            return self.cursor.execute(sql, params)
          File "/home/evennia/muddev/evenv/lib/python3.7/site-packages/django/db/utils.py", line 89, in __exit__
            raise dj_exc_value.with_traceback(traceback) from exc_value
          File "/home/evennia/muddev/evenv/lib/python3.7/site-packages/django/db/backends/utils.py", line 84, in _execute
            return self.cursor.execute(sql, params)
          File "/home/evennia/muddev/evenv/lib/python3.7/site-packages/django/db/backends/mysql/base.py", line 71, in execute
            return self.cursor.execute(query, args)
          File "/home/evennia/muddev/evenv/lib/python3.7/site-packages/MySQLdb/cursors.py", line 206, in execute
            res = self._query(query)
          File "/home/evennia/muddev/evenv/lib/python3.7/site-packages/MySQLdb/cursors.py", line 319, in _query
            db.query(q)
          File "/home/evennia/muddev/evenv/lib/python3.7/site-packages/MySQLdb/connections.py", line 259, in query
            _mysql.connection.query(self, query)
        django.db.utils.OperationalError: (2006, 'MySQL server has gone away')
```

#### Expected behavior

If nothing else, rather than bombing out I would hope the portal would re-establish a mysql connection.  The mysqld is there and waiting.

Also, evennia isn't handling restoring service very easily - an "evennia restart" doesn't resolve the situation - I have to "evennia stop" and "evennia start" to get the portal back properly.

#### Environment, Evennia version, OS etc

evennia 0.9.5, commit 135e1ba14de4ed8b3dd5a26dc57ebbb79eebf99a from Feb. 27 
mariadb 10.3.27 - wait_timeout is 28800

I have tried "CONN_MAX_AGE = 3600" in mygame/server/conf/settings.py but it hasn't improved the issue.



Discussion:
I happen to be using MySQL and do not have this problem as of late-- but I used to.

What changed for me is that these days I happen to have a script running that interacts with the db once per minute. Since there is constant activity, it never times out anymore-- I forgot this was even a problem.

As a workaround, try creating a persistent script that does something menial like counting the objects in your db or does `ObjectDB.objects.all().exists()` every 60 seconds or so.
Evennia has a timer running both in the portal and server processes that will refresh the database connection every 7 hours (default MySQL timeout is 8 hours as far as I understand):

- https://github.com/evennia/evennia/blob/master/evennia/server/server.py#L152
- https://github.com/evennia/evennia/blob/master/evennia/server/portal/portal.py#L122

So it's strange that this issue is still a thing. :confused: 
I'm confused about both of the functions that this code resides in.  The first comment in _server_maintenance() states:

"This maintenance function handles repeated checks and updates that
    the server needs to do. It is called every minute."

But it appears that the time calculations for each item, both in server.py and portal.py, suggest the calculations are in seconds:

if _MAINTENANCE_COUNT % 300 == 0:  # check cache size every 5 minutes
if _MAINTENANCE_COUNT % 3600 == 0: # validate scripts every hour

etc.

In doing some testing, it appears that this function is only being called once per minute, hence _MAINTENANCE_COUNT is only being incremented once per minute.  So, it appears that all of the maintenance functions are off by a factor of 60, the 5 minute check being run once every 5 hours, the script validation only running once every 60 hours, and, our main issue function, the check to close the DB connection every 7 hours is really only triggering once every 17.5 days.

I think all of these calculations in both server.py and in _portal_maintenance() in portal.py are going to have to be adjusted to not use 3600 as calculation for hours, but 60, instead, and the 300 being set to 5.

I am a relative python newbie, so please take anything I'm saying with the understanding that I could well be overlooking something else.  However, I added in print statements to show me how often items are triggering when I changed time values, and it all seems to line up.  In portal.py, I used:

```
    _MAINTENANCE_COUNT += 1
    print ("_MAINTENANCE_COUNT = " + str(_MAINTENANCE_COUNT))
```

And the logs are updating every minute, not second, showing me:

```
21-03-27 21:21:14  |Portal| _MAINTENANCE_COUNT = 83
21-03-27 21:22:14  |Portal| _MAINTENANCE_COUNT = 84
21-03-27 21:23:14  |Portal| _MAINTENANCE_COUNT = 85
21-03-27 21:24:14  |Portal| _MAINTENANCE_COUNT = 86
21-03-27 21:25:14  |Portal| _MAINTENANCE_COUNT = 87
```


Resolving patch:
diff --git a/evennia/server/portal/portal.py b/evennia/server/portal/portal.py
--- a/evennia/server/portal/portal.py
+++ b/evennia/server/portal/portal.py
@@ -119,7 +119,7 @@ def _portal_maintenance():
 
     _MAINTENANCE_COUNT += 1
 
-    if _MAINTENANCE_COUNT % (3600 * 7) == 0:
+    if _MAINTENANCE_COUNT % (60 * 7) == 0:
         # drop database connection every 7 hrs to avoid default timeouts on MySQL
         # (see https://github.com/evennia/evennia/issues/1376)
         connection.close()
diff --git a/evennia/server/server.py b/evennia/server/server.py
--- a/evennia/server/server.py
+++ b/evennia/server/server.py
@@ -140,16 +140,16 @@ def _server_maintenance():
     _GAMETIME_MODULE.SERVER_RUNTIME_LAST_UPDATED = now
     ServerConfig.objects.conf("runtime", _GAMETIME_MODULE.SERVER_RUNTIME)
 
-    if _MAINTENANCE_COUNT % 300 == 0:
+    if _MAINTENANCE_COUNT % 5 == 0:
         # check cache size every 5 minutes
         _FLUSH_CACHE(_IDMAPPER_CACHE_MAXSIZE)
-    if _MAINTENANCE_COUNT % 3600 == 0:
+    if _MAINTENANCE_COUNT % 60 == 0:
         # validate scripts every hour
         evennia.ScriptDB.objects.validate()
-    if _MAINTENANCE_COUNT % 3700 == 0:
+    if _MAINTENANCE_COUNT % 61 == 0:
         # validate channels off-sync with scripts
         evennia.CHANNEL_HANDLER.update()
-    if _MAINTENANCE_COUNT % (3600 * 7) == 0:
+    if _MAINTENANCE_COUNT % (60 * 7) == 0:
         # drop database connection every 7 hrs to avoid default timeouts on MySQL
         # (see https://github.com/evennia/evennia/issues/1376)
         connection.close()


Test patch:
diff --git a/evennia/server/tests/test_server.py b/evennia/server/tests/test_server.py
--- a/evennia/server/tests/test_server.py
+++ b/evennia/server/tests/test_server.py
@@ -47,7 +47,7 @@ def test__server_maintenance_flush(self):
             _FLUSH_CACHE=DEFAULT,
             connection=DEFAULT,
             _IDMAPPER_CACHE_MAXSIZE=1000,
-            _MAINTENANCE_COUNT=600 - 1,
+            _MAINTENANCE_COUNT=5 - 1,
             ServerConfig=DEFAULT,
         ) as mocks:
             mocks["connection"].close = MagicMock()
@@ -65,7 +65,7 @@ def test__server_maintenance_validate_scripts(self):
             _FLUSH_CACHE=DEFAULT,
             connection=DEFAULT,
             _IDMAPPER_CACHE_MAXSIZE=1000,
-            _MAINTENANCE_COUNT=3600 - 1,
+            _MAINTENANCE_COUNT=60 - 1,
             _LAST_SERVER_TIME_SNAPSHOT=0,
             ServerConfig=DEFAULT,
         ) as mocks:
@@ -84,7 +84,7 @@ def test__server_maintenance_channel_handler_update(self):
             _FLUSH_CACHE=DEFAULT,
             connection=DEFAULT,
             _IDMAPPER_CACHE_MAXSIZE=1000,
-            _MAINTENANCE_COUNT=3700 - 1,
+            _MAINTENANCE_COUNT=61 - 1,
             _LAST_SERVER_TIME_SNAPSHOT=0,
             ServerConfig=DEFAULT,
         ) as mocks:
@@ -102,7 +102,7 @@ def test__server_maintenance_close_connection(self):
             _FLUSH_CACHE=DEFAULT,
             connection=DEFAULT,
             _IDMAPPER_CACHE_MAXSIZE=1000,
-            _MAINTENANCE_COUNT=(3600 * 7) - 1,
+            _MAINTENANCE_COUNT=(60 * 7) - 1,
             _LAST_SERVER_TIME_SNAPSHOT=0,
             ServerConfig=DEFAULT,
         ) as mocks: