Repository: edgedb/edgedb

Issue:
discussion: handle new tasks on server shutdown
For new tasks created after the server is requessted to shutdown and before the loop actually stops, we have 3 options to handle them:

1. Don't even create the `coroutine` object to avoid the `RuntimeWarning` (this PR). The downside is the inconsistent `server.create_task()` API that takes a coroutine factory rather than a coroutine object.
2. Don't call `create_task()` when `server._accept_new_tasks == False`. This adds a lot of duplicate checks before calling `server.create_task()`.
3. Create the tasks anyways, and carefully check every task and modify them to honor the shutdown event. We don't even need the "interruptable" tasks in this case. The risk is, the server may freeze at exit if in the future someone adds new tasks that cannot clean themselves at server exit.
  * We could use `SignalController().wait_for(coro, cancel_on=SIGxxx)` in most tasks, but this also means we'll have a lot of signal-triggered cancellation.
  * At the same time, certain server-exit sanity checks in the tasks are unavoidable. Even if we choose other options, these checks are also necessary for "un-interruptable" tasks to stop, because those tasks may have already started before the server is requested to stop.

Mute coroutine warning at shutdown by immediate cancellation
After revisiting #3112, I found a more reliable option 4 to address the "coroutine ... was never awaited" warning - cancel the late tasks immediately after creating them.

Closes #3112
Warning during bootstrap
- EdgeDB Version: 1.0-dev.6229+004227f

Warns during bootstrap:
```
Successfully installed 1.0-dev.6229+004227f
Initializing EdgeDB instance...
WARNING 268 2021-11-27T16:25:05.691 py.warnings: /home/user1/.local/share/edgedb/portable/1.0-dev.6229/lib/python3.10/site-packages/edb/server/server.py:1288: RuntimeWarning: coroutine '_signal_sysevent' was never awaited
  await binary.EdgeConnection.run_script(
```


Discussion:

Can we prevent the task from starting instead?
For the case #3218, I would prefer doing `if not self.server.shutting_down()` in `signal_side_effects`. Because in this case we explicitly know that these side effects are irrelevant after shutdown. But I'm not sure about other (future) tasks.
Yeah, that's exactly option 2 in #3112. We may need both option 2 & 4 for both explicitness and correctness.
> Yeah, that's exactly option 2 in #3112. We may need both option 2 & 4 for both explicitness and correctness.

Do we have too many of such tasks? I think I have only seen that one in #3218 so far. Also there is no option for so I'm not sure what you mean.
> Do we have too many of such tasks?

There're a few, but not too many. I've pushed [a fix](https://github.com/edgedb/edgedb/pull/3220/commits/55d192c904938f671be5dafa4298322d8ea0809e) to add a check to most of them.

> > Yeah, that's exactly option 2 in #3112. We may need both option 2 & 4 for both explicitness and correctness.
>
>  Also there is no option for so I'm not sure what you mean.

Oh I mean this in #3112 description:

> 2. Don't call `create_task()` when `server._accept_new_tasks == False`. This adds a lot of duplicate checks before calling `server.create_task()`.

> I would prefer `if not self.is_shutting_down()` or something similar. `_accept_new_tasks` doesn't say enough. Like does it temporary not accepting tasks? Should we retry in a timeout? Or maybe it doesn't accept because it is starting up?

Well `_accept_new_tasks` is not 100% `!is_shutting_down()`, particularly in e.g. `Server.run_startup_script_and_exit()` before `Server.start()` is called. I've renamed `_accept_new_tasks` to `_shutting_down`, and use `Server._task_group is None` in `Server.create_task()` to check if we're allowed to create tasks. This should cover all cases.
At the moment, internally the `Server` has 4 states:
* `_initing=True`: while `Server.init()` is running
* `_serving=True`: since the TCP socket started to accept connections, before `Server.stop()` is called
* `_shutting_down=True`: after `Server.stop()` is called
* In the rest of the time, the server stays in a "default" state.

Bootstrap is like: `default` -> `initing` -> `default` (run_startup_script_and_exit)
Server from start to stop: `default` -> `initing` -> `default` (startup_script) -> `serving` -> `shutting_down`

* The server will abort new client connections immediately if not `serving`
* The server will not trigger `__edgedb_sysevent__` in `shutting_down` and `default` because the `sys_pgcon` is likely closing. This should be fixed I think to always trigger the event, but not handling the notification.
@fantix, from your description I'm still confused why we need `_discard_coro`? If it's just in case "we forgot something", let's make warnings visible at least in dev mode (although, since nobody were concerned about these warnings previously, until we saw them on installation, it maybe not enough).
I'm also not convinced of the utility of `_discard_coro`.  We are trading Python's standard "coroutine not awaited" for a custom and lower visibility "task ignored".  An explicit check in `signal_side_effects` should be enough for now.
> I'm also not convinced of the utility of _discard_coro.

FWIW that function is one reason why I don't want to merge this PR.
should be fixed together with #3112 
Then #3112 should be prioritized I think. Seeing this warning each time you create an instance on 1.0 would be unfortunate. Or maybe we can come up with a simpler fix for just this warning?

Resolving patch:
diff --git a/edb/server/server.py b/edb/server/server.py
--- a/edb/server/server.py
+++ b/edb/server/server.py
@@ -557,7 +557,7 @@ async def reload_sys_config(self):
 
     def schedule_reported_config_if_needed(self, setting_name):
         setting = self._config_settings[setting_name]
-        if setting.report:
+        if setting.report and self._accept_new_tasks:
             self.create_task(
                 self.load_reported_config(), interruptable=True)
 
@@ -1106,6 +1106,9 @@ async def _signal_sysevent(self, event, **kwargs):
             raise
 
     def _on_remote_ddl(self, dbname):
+        if not self._accept_new_tasks:
+            return
+
         # Triggered by a postgres notification event 'schema-changes'
         # on the __edgedb_sysevent__ channel
         async def task():
@@ -1118,6 +1121,9 @@ async def task():
         self.create_task(task(), interruptable=True)
 
     def _on_remote_database_config_change(self, dbname):
+        if not self._accept_new_tasks:
+            return
+
         # Triggered by a postgres notification event 'database-config-changes'
         # on the __edgedb_sysevent__ channel
         async def task():
@@ -1131,6 +1137,9 @@ async def task():
         self.create_task(task(), interruptable=True)
 
     def _on_local_database_config_change(self, dbname):
+        if not self._accept_new_tasks:
+            return
+
         # Triggered by DB Index.
         # It's easier and safer to just schedule full re-introspection
         # of the DB and update all components of it.
@@ -1145,6 +1154,9 @@ async def task():
         self.create_task(task(), interruptable=True)
 
     def _on_remote_system_config_change(self):
+        if not self._accept_new_tasks:
+            return
+
         # Triggered by a postgres notification event 'system-config-changes'
         # on the __edgedb_sysevent__ channel
 
@@ -1159,6 +1171,9 @@ async def task():
         self.create_task(task(), interruptable=True)
 
     def _on_global_schema_change(self):
+        if not self._accept_new_tasks:
+            return
+
         async def task():
             try:
                 await self._reintrospect_global_schema()
@@ -1187,7 +1202,10 @@ def _on_sys_pgcon_connection_lost(self, exc):
             )
             self.__sys_pgcon = None
             self._sys_pgcon_ready_evt.clear()
-            self.create_task(self._reconnect_sys_pgcon(), interruptable=True)
+            if self._accept_new_tasks:
+                self.create_task(
+                    self._reconnect_sys_pgcon(), interruptable=True
+                )
             self._on_pgcon_broken(True)
         except Exception:
             metrics.background_errors.inc(1.0, 'on_sys_pgcon_connection_lost')
@@ -1572,17 +1590,16 @@ def create_task(self, coro, *, interruptable):
         # randomly in the middle when the event loop stops; while tasks with
         # interruptable=False are always awaited before the server stops, so
         # that e.g. all finally blocks get a chance to execute in those tasks.
+        # Therefore, it is an error trying to create a task while the server is
+        # not expecting one, so always couple the call with an additional check
         if self._accept_new_tasks:
             if interruptable:
                 return self.__loop.create_task(coro)
             else:
                 return self._task_group.create_task(coro)
         else:
-            # Silence the "coroutine not awaited" warning
-            logger.debug(
-                "Task is not started and ignored: %r", coro.cr_code.co_name
-            )
-            coro.__await__()
+            # Hint: add `if server._accept_new_tasks` before `.create_task()`
+            raise RuntimeError("task cannot be created at this time")
 
     async def serve_forever(self):
         await self._stop_evt.wait()
@@ -1622,9 +1639,10 @@ def on_switch_over(self):
         # to the old master.
         self._ha_master_serial += 1
 
-        self.create_task(
-            self._pg_pool.prune_all_connections(), interruptable=True
-        )
+        if self._accept_new_tasks:
+            self.create_task(
+                self._pg_pool.prune_all_connections(), interruptable=True
+            )
 
         if self.__sys_pgcon is None:
             # Assume a reconnect task is already running, now that we know the


Test patch: