Repository: edgedb/edgedb

Issue:
Azure maintenance, edgedb failed to reconnect to backend Postgres db
<!-- Please search existing issues to avoid creating duplicates. -->

<!--
For the EdgeDB Version: run `edgedb query 'select sys::get_version_as_str()'` from your project directory (or run `select sys::get_version_as_str();` in the EdgeDB interactive shell).
For the EdgeDB CLI Version: Run `edgedb --version` from anywhere
-->

- EdgeDB Version: {'4.2+9856f3d'}
- EdgeDB CLI Version: EdgeDB CLI 4.0.2+80e48a7
- OS Version: node:18-bullseye-slim (base image)

Steps to Reproduce:

1. Deployed to azure following edgedb guide https://www.edgedb.com/docs/guides/deployment/azure_flexibleserver
2. Azure had a maintenance event where the backing Postgres server was brought down and back up.
3. Edgedb container failed to reconnect to the backing Postgres server when it was made available again. With the error:
```
ERROR 14 2023-12-03T06:29:41.248 edb.server: Connection to the system database is closed.
---- Exception occurred: could not create directory "pg_tblspc/16386/PG_15_202209061/31817": No such file or directory ----

1. edb.server.pgcon.errors.BackendError: could not create directory "pg_tblspc/16386/PG_15_202209061/31817": No such file or directory

---- Traceback ----

    /usr/lib/x86_64-linux-gnu/edgedb-server-4/lib/python3.11/site-packages/edb/server/tenant.py, line 585, in _reconnect_sys_pgcon
        > conn = await self._pg_connect(defines.EDGEDB_SYSTEM_DB)
    /usr/lib/x86_64-linux-gnu/edgedb-server-4/lib/python3.11/site-packages/edb/server/tenant.py, line 438, in _pg_connect
        > rv = await pgcon.connect(
    edb/server/pgcon/pgcon.pyx, line 385, in connect
        
    edb/server/pgcon/pgcon.pyx, line 1580, in sql_execute
        
    edb/server/pgcon/pgcon.pyx, line 1566, in _sql_execute
        

edb.server.pgcon.errors.BackendError: could not create directory "pg_tblspc/16386/PG_15_202209061/31817": No such file or directory
ERROR 14 2023-12-03T06:31:06.628 asyncio: Task exception was never retrieved
future: <Task finished name='Task-3225' coro=<Tenant._reconnect_sys_pgcon() done, defined at /usr/lib/x86_64-linux-gnu/edgedb-server-4/lib/python3.11/site-packages/edb/server/tenant.py:580> exception=BackendError('could not create directory "pg_tblspc/16386/PG_15_202209061/31817": No such file or directory')>
```


The error shown from the application container where edgedb cli tool is installed was:
```
edgedb error: BackendUnavailableError: cannot fetch database version: Postgres is not available: terminating connection due to administrator command
  Server traceback:
      Traceback (most recent call last):
        File "edb/server/protocol/binary.pyx", line 959, in edb.server.protocol.binary.EdgeConnection.main_step
        File "edb/server/protocol/binary.pyx", line 910, in execute
        File "edb/server/protocol/binary.pyx", line 704, in _execute
        File "edb/server/protocol/frontend.pyx", line 133, in get_pgcon
        File "edb/server/protocol/frontend.pyx", line 127, in edb.server.protocol.frontend.FrontendConnection.get_pgcon
        File "/usr/lib/x86_64-linux-gnu/edgedb-server-4/lib/python3.11/site-packages/edb/server/tenant.py", line 661, in acquire_pgcon
          raise errors.BackendUnavailableError(
      edb.errors.BackendUnavailableError: Postgres is not available: terminating connection due to administrator command
```

The maintenance event from azure details are below:
![image](https://github.com/edgedb/edgedb/assets/147151862/1b0ce092-75a8-46a0-9e5e-7d1779526d4f)


The issue was fixed at the time by restarting the edgedb container.

The expectation is that edgedb should be able to gracefully handful a similar maintenance event and reconnect automatically when the backing Postgres database becomes re-available. 



Discussion:
@fantix, it seems we are giving up on attempts to reconnect the the backend too easily:

https://github.com/edgedb/edgedb/blob/7601ba076c4e0ef810f6239a7b0238f6a4b6e056/edb/server/tenant.py#L618-L624

Is there a reason why we are not retrying on all backend errors?
I think the reason was to rely on a failover event, but it doesn't make much sense to me now. The conditions here are fine, I'll change the `raise` below into a log for backend errors that we don't recognize, so that the retry loop can go on.

Resolving patch:
diff --git a/edb/server/tenant.py b/edb/server/tenant.py
--- a/edb/server/tenant.py
+++ b/edb/server/tenant.py
@@ -604,17 +604,19 @@ async def _reconnect_sys_pgcon(self) -> None:
         try:
             conn = None
             while self._running:
+                # Keep retrying as far as:
+                #   1. This tenant is still running
+                #   2. We still cannot connect to the Postgres cluster
                 try:
                     conn = await self._pg_connect(defines.EDGEDB_SYSTEM_DB)
                     break
                 except OSError:
-                    # Keep retrying as far as:
-                    #   1. This tenant is still running,
-                    #   2. We still cannot connect to the Postgres cluster, or
                     pass
                 except pgcon_errors.BackendError as e:
-                    #   3. The Postgres cluster is still starting up, or the
-                    #      HA failover is still in progress
+                    # Be quiet if the Postgres cluster is still starting up,
+                    # or the HA failover is still in progress.
+                    # TODO: ERROR_FEATURE_NOT_SUPPORTED should be removed
+                    # once PostgreSQL supports SERIALIZABLE in hot standbys
                     if not (
                         e.code_is(pgcon_errors.ERROR_FEATURE_NOT_SUPPORTED)
                         or e.code_is(pgcon_errors.ERROR_CANNOT_CONNECT_NOW)
@@ -622,11 +624,10 @@ async def _reconnect_sys_pgcon(self) -> None:
                             pgcon_errors.ERROR_READ_ONLY_SQL_TRANSACTION
                         )
                     ):
-                        # TODO: ERROR_FEATURE_NOT_SUPPORTED should be removed
-                        # once PostgreSQL supports SERIALIZABLE in hot standbys
-                        raise
+                        logger.error("Failed connecting to the backend: %s", e)
 
                 if self._running:
+                    logger.info("Waiting for the backend to recover")
                     try:
                         # Retry after INTERVAL seconds, unless the event is set
                         # and we can retry immediately after the event.


Test patch: