Repository: PrefectHQ/prefect

Issue:
Slow postgres query when filtering logs
### First check

- [X] I added a descriptive title to this issue.
- [X] I used the GitHub search to find a similar issue and didn't find it.
- [X] I searched the Prefect documentation for this issue.
- [X] I checked that this issue is related to Prefect and not one of its dependencies.

### Bug summary

When the prefect database contains a certain amount of logs, the API endpoints that produces SQL queries to the `log` table are slow and we then can hit timeouts at the proxy layer or at the API layer depending on the [PREFECT_API_DATABASE_TIMEOUT](https://docs.prefect.io/api-ref/prefect/settings/#prefect.settings.PREFECT_API_DATABASE_TIMEOUT) setting configured.

This particular slow query could possibly been explained by a postgres behavior when using `ORDER BY` and a small `LIMIT` at the same time. More info [here](https://stackoverflow.com/a/21386282) and [here](https://dba.stackexchange.com/a/130280).

We can solve that issue by either increasing the LIMIT or by creating an index on the two columns `flow_run_id` and `timestamp` (in addition to the already existing indexes on both columns) of the table `log`:
```sql
CREATE INDEX idx_log_flow_run_id_timestamp ON log (flow_run_id, timestamp);
```

Query execution time (calculated with `\timing on` postgres option):
- Without the index: `Time: 258844.350 ms (04:18.844)`
- With the index: `Time: 43.679 ms`

### Reproduction

- log table size: `~12GB`

- API endpoint tested: `/api/logs/filter`

- Endpoint payload tested: 
```json
{"logs":{"level":{"ge_":0},"flow_run_id":{"any_":["3330bca5-bf56-4144-9c65-5d80f5789d12"]}},"sort":"TIMESTAMP_ASC","offset":0,"limit":200}
```

- SQL query tested:
```sql
SELECT log.id, log.created, log.updated, log.name, log.level, log.flow_run_id, log.task_run_id, log.message, log.timestamp FROM log
WHERE log.level >= 0::integer AND log.flow_run_id IN ('3330bca5-bf56-4144-9c65-5d80f5789d12'::uuid)
ORDER BY log.timestamp ASC
LIMIT 200::integer
OFFSET 0::integer;
```


### Error (not fully relevant here but that's how we detected the DB query was slow)

```python3
Exception in ASGI application
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/uvicorn/protocols/http/h11_impl.py", line 429, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
  File "/usr/local/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 78, in __call__
    return await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/uvicorn/middleware/message_logger.py", line 86, in __call__
    raise exc from None
  File "/usr/local/lib/python3.10/site-packages/uvicorn/middleware/message_logger.py", line 82, in __call__
    await self.app(scope, inner_receive, inner_send)
  File "/usr/local/lib/python3.10/site-packages/fastapi/applications.py", line 276, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/applications.py", line 122, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/middleware/errors.py", line 184, in __call__
    raise exc
  File "/usr/local/lib/python3.10/site-packages/starlette/middleware/errors.py", line 162, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.10/site-packages/starlette/middleware/cors.py", line 92, in __call__
    await self.simple_response(scope, receive, send, request_headers=headers)
  File "/usr/local/lib/python3.10/site-packages/starlette/middleware/cors.py", line 147, in simple_response
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 79, in __call__
    raise exc
  File "/usr/local/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 68, in __call__
    await self.app(scope, receive, sender)
  File "/usr/local/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in __call__
    raise e
  File "/usr/local/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 718, in __call__
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 443, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/fastapi/applications.py", line 276, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/applications.py", line 122, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/middleware/errors.py", line 184, in __call__
    raise exc
  File "/usr/local/lib/python3.10/site-packages/starlette/middleware/errors.py", line 162, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 79, in __call__
    raise exc
  File "/usr/local/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 68, in __call__
    await self.app(scope, receive, sender)
  File "/usr/local/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in __call__
    raise e
  File "/usr/local/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 718, in __call__
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 276, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 66, in app
    response = await func(request)
  File "/usr/local/lib/python3.10/site-packages/prefect/server/utilities/server.py", line 103, in handle_response_scoped_depends
    response = await default_handler(request)
  File "/usr/local/lib/python3.10/site-packages/fastapi/routing.py", line 237, in app
    raw_response = await run_endpoint_function(
  File "/usr/local/lib/python3.10/site-packages/fastapi/routing.py", line 163, in run_endpoint_function
    return await dependant.call(**values)
  File "/usr/local/lib/python3.10/site-packages/prefect/server/api/logs.py", line 42, in read_logs
    return await models.logs.read_logs(
  File "/usr/local/lib/python3.10/site-packages/prefect/server/database/dependencies.py", line 119, in async_wrapper
    return await fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/prefect/server/models/logs.py", line 76, in read_logs
    result = await session.execute(query)
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/ext/asyncio/session.py", line 214, in execute
    result = await greenlet_spawn(
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 126, in greenlet_spawn
    result = context.throw(*sys.exc_info())
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 1714, in execute
    result = conn._execute_20(statement, params or {}, execution_options)
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20
    return meth(self, args_10style, kwargs_10style, execution_options)
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection
    return connection._execute_clauseelement(
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement
    ret = self._execute_context(
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context
    self._handle_dbapi_exception(
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2128, in _handle_dbapi_exception
    util.raise_(exc_info[1], with_traceback=exc_info[2])
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/util/compat.py", line 211, in raise_
    raise exception
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context
    self.dialect.do_execute(
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
    cursor.execute(statement, parameters)
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 479, in execute
    self._adapt_connection.await_(
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 68, in await_only
    return current.driver.switch(awaitable)
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 121, in greenlet_spawn
    value = await result
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 454, in _prepare_and_execute
    self._handle_exception(error)
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 389, in _handle_exception
    self._adapt_connection._handle_exception(error)
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 684, in _handle_exception
    raise error
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 442, in _prepare_and_execute
    self._rows = await prepared_stmt.fetch(*parameters)
  File "/usr/local/lib/python3.10/site-packages/asyncpg/prepared_stmt.py", line 176, in fetch
    data = await self.__bind_execute(args, 0, timeout)
  File "/usr/local/lib/python3.10/site-packages/asyncpg/prepared_stmt.py", line 241, in __bind_execute
    data, status, _ = await self.__do_execute(
  File "/usr/local/lib/python3.10/site-packages/asyncpg/prepared_stmt.py", line 230, in __do_execute
    return await executor(protocol)
  File "asyncpg/protocol/protocol.pyx", line 201, in bind_execute
asyncio.exceptions.TimeoutError
```


### Versions

```Text
PREFECT

Version:             2.8.7
API version:         0.8.4
Python version:      3.10.10
Git commit:          a6d6c6fc
Built:               Thu, Mar 23, 2023 3:27 PM
OS/Arch:             linux/x86_64
Profile:             default
Server type:         server

POSTGRES

Version: PostgreSQL 14
Type   : GCP Cloud SQL
```


### Additional context

I think other endpoints suffer from the same problems like `/api/flow_runs/filter`.


Discussion:
Thanks John! It sounds reasonable to add an index if you're willing to do so. 

Resolving patch:
diff --git a/src/prefect/server/database/migrations/versions/postgresql/2023_04_04_132534_3bf47e3ce2dd_add_index_on_log.py b/src/prefect/server/database/migrations/versions/postgresql/2023_04_04_132534_3bf47e3ce2dd_add_index_on_log.py
new file mode 100644
--- /dev/null
+++ b/src/prefect/server/database/migrations/versions/postgresql/2023_04_04_132534_3bf47e3ce2dd_add_index_on_log.py
@@ -0,0 +1,27 @@
+"""add_index_on_log
+
+Revision ID: 3bf47e3ce2dd
+Revises: 46bd82c6279a
+Create Date: 2023-04-04 13:25:34.694078
+
+"""
+from alembic import op
+
+# revision identifiers, used by Alembic.
+revision = "3bf47e3ce2dd"
+down_revision = "46bd82c6279a"
+branch_labels = None
+depends_on = None
+
+
+def upgrade():
+    op.create_index(
+        "ix_log__flow_run_id_timestamp",
+        "log",
+        ["flow_run_id", "timestamp"],
+        unique=False,
+    )
+
+
+def downgrade():
+    op.drop_index("ix_log__flow_run_id_timestamp", table_name="log")
diff --git a/src/prefect/server/database/migrations/versions/sqlite/2023_04_04_115150_553920ec20e9_add_index_on_log.py b/src/prefect/server/database/migrations/versions/sqlite/2023_04_04_115150_553920ec20e9_add_index_on_log.py
new file mode 100644
--- /dev/null
+++ b/src/prefect/server/database/migrations/versions/sqlite/2023_04_04_115150_553920ec20e9_add_index_on_log.py
@@ -0,0 +1,34 @@
+"""add_index_on_log
+
+Revision ID: 553920ec20e9
+Revises: 422f8ba9541d
+Create Date: 2023-04-04 11:51:50.514999
+
+"""
+from alembic import op
+
+# revision identifiers, used by Alembic.
+revision = "553920ec20e9"
+down_revision = "422f8ba9541d"
+branch_labels = None
+depends_on = None
+
+
+def upgrade():
+    op.execute("PRAGMA foreign_keys=OFF")
+
+    with op.batch_alter_table("log", schema=None) as batch_op:
+        batch_op.create_index(
+            "ix_log__flow_run_id_timestamp", ["flow_run_id", "timestamp"], unique=False
+        )
+
+    op.execute("PRAGMA foreign_keys=ON")
+
+
+def downgrade():
+    op.execute("PRAGMA foreign_keys=OFF")
+
+    with op.batch_alter_table("log", schema=None) as batch_op:
+        batch_op.drop_index("ix_log__flow_run_id_timestamp")
+
+    op.execute("PRAGMA foreign_keys=ON")
diff --git a/src/prefect/server/database/orm_models.py b/src/prefect/server/database/orm_models.py
--- a/src/prefect/server/database/orm_models.py
+++ b/src/prefect/server/database/orm_models.py
@@ -924,6 +924,16 @@ class ORMLog:
     # The client-side timestamp of this logged statement.
     timestamp = sa.Column(Timestamp(), nullable=False, index=True)
 
+    @declared_attr
+    def __table_args__(cls):
+        return (
+            sa.Index(
+                "ix_log__flow_run_id_timestamp",
+                "flow_run_id",
+                "timestamp",
+            ),
+        )
+
 
 @declarative_mixin
 class ORMConcurrencyLimit:


Test patch: