Repository: paperless-ngx/paperless-ngx

Issue:
[BUG] "Shared By Me" filter causes error with > 1 user / group in PostgreSQL environment
### Description

When filtering documents by "Permissions -> Shared By Me" an error 500 message is shown.

The setup is using PostgreSQL 15.5 but installed as a standalone RedHat Enterprise Linux package.


### Steps to reproduce

1. Go to "Documents"
2. Filter by "Permissions -> Shared By Me"
3. An error popup is displayed showing an error 500 error msg.

### Webserver logs

```bash
[2024-01-14 13:18:27,014] [ERROR] [django.request] Internal Server Error: /api/documents/
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
psycopg2.errors.CardinalityViolation: more than one row returned by a subquery used as an expression


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/asgiref/sync.py", line 534, in thread_handler
    raise exc_info[1]
  File "/usr/local/lib/python3.11/site-packages/django/core/handlers/exception.py", line 42, in inner
    response = await get_response(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/asgiref/sync.py", line 534, in thread_handler
    raise exc_info[1]
  File "/usr/local/lib/python3.11/site-packages/django/core/handlers/base.py", line 253, in _get_response_async
    response = await wrapped_callback(
               ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/asgiref/sync.py", line 479, in __call__
    ret: _R = await loop.run_in_executor(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/asgiref/current_thread_executor.py", line 40, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/asgiref/sync.py", line 538, in thread_handler
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/views/decorators/csrf.py", line 56, in wrapper_view
    return view_func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/rest_framework/viewsets.py", line 125, in view
    return self.dispatch(request, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/rest_framework/views.py", line 509, in dispatch
    response = self.handle_exception(exc)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/rest_framework/views.py", line 469, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/usr/local/lib/python3.11/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
    raise exc
  File "/usr/local/lib/python3.11/site-packages/rest_framework/views.py", line 506, in dispatch
    response = handler(request, *args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/paperless/src/documents/views.py", line 738, in list
    return super().list(request)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/rest_framework/mixins.py", line 40, in list
    page = self.paginate_queryset(queryset)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/rest_framework/generics.py", line 171, in paginate_queryset
    return self.paginator.paginate_queryset(queryset, self.request, view=self)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/rest_framework/pagination.py", line 204, in paginate_queryset
    self.page = paginator.page(page_number)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/paginator.py", line 72, in page
    number = self.validate_number(number)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/paginator.py", line 53, in validate_number
    if number > self.num_pages:
                ^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/utils/functional.py", line 57, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
                                         ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/paginator.py", line 99, in num_pages
    if self.count == 0 and not self.allow_empty_first_page:
       ^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/utils/functional.py", line 57, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
                                         ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/paginator.py", line 93, in count
    return c()
           ^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 608, in count
    return self.query.get_count(using=self.db)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/models/sql/query.py", line 568, in get_count
    return obj.get_aggregation(using, {"__count": Count("*")})["__count"]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/models/sql/query.py", line 554, in get_aggregation
    result = compiler.execute_sql(SINGLE)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/models/sql/compiler.py", line 1562, in execute_sql
    cursor.execute(sql, params)
  File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers
    return executor(sql, params, many, context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 84, in _execute
    with self.db.wrap_database_errors:
  File "/usr/local/lib/python3.11/site-packages/django/db/utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
django.db.utils.ProgrammingError: more than one row returned by a subquery used as an expression
```


### Browser logs

_No response_

### Paperless-ngx version

2.3.3

### Host OS

Rocky Linux 8.9

### Installation method

Docker - official image

### Browser

Any

### Configuration changes

The PostgreSQL database is a standalone v15.5 setup.

### Other

_No response_

### Please confirm the following

- [X] I believe this issue is a bug that affects all users of Paperless-ngx, not something specific to my installation.
- [X] I have already searched for relevant existing issues and discussions before opening this report.
- [X] I have updated the title field above with a concise description.


Discussion:
I have no idea how to reproduce this. Unsurprisingly, the steps above are not enough
Ok, I bisected this issue and here is why it fails. 

This is a stripped down version of the query that fails:
```
SELECT DISTINCT
  "documents_document"."id",
  COUNT((SELECT U0."group_id" FROM "guardian_groupobjectpermission" U0 WHERE (U0."content_type_id" = 11 AND U0."object_pk" = ("documents_document"."id")::varchar))) AS "num_shared_groups" 
FROM "documents_document"
GROUP BY "documents_document"."id"
LIMIT 5;
```
The reason why it fails is because PostgreSQL does not allow multiple results in a subquery of a SELECT statement. The COUNT() here is too late.

The query would work when changed to this:
```
SELECT DISTINCT
  "documents_document"."id",
  (SELECT count(U0."group_id") FROM "guardian_groupobjectpermission" U0 WHERE (U0."content_type_id" = 11 AND U0."object_pk" = ("documents_document"."id")::varchar)) AS "num_shared_groups" 
FROM "documents_document"
GROUP BY "documents_document"."id" LIMIT 5;

 id | num_shared_groups 
----+-------------------
  1 |                 3
  2 |                 3
  3 |                 3
  4 |                 3
  5 |                 3
(5 rows)
```

Thanks that’s helpful, I will dig into it as soon as I have a sec. Still not exactly clear what conditions are needed but just looking at the above on my phone, maybe it’s when an object is shared with multiple groups
> maybe it’s when an object is shared with multiple groups

Yes, that could be it. I have an "Editors" and a "Viewers" group. As editors are automatically assigned to the the view permission as well, I guess that's why the count is 3.

Also I noted in the query log, that the above way of counting is used for more SQL queries. So I guess this error can happen for other scenarios as well.

Its really not clear to me how you would reverse-engineer the above SQL query back to where its actually constructed:

https://github.com/paperless-ngx/paperless-ngx/blob/088803d52831d9ef911d493063723fac46e0ee48/src/documents/filters.py#L129-L136

Edit: Ok, confirmed that this only happens with Postgres and with > 1 user / groups. But still not clear how to fix, quite tricky because the query really is doing exactly what we want... 🤔

Appreciate thoughts if anyone else can help

Resolving patch:
diff --git a/src/documents/filters.py b/src/documents/filters.py
--- a/src/documents/filters.py
+++ b/src/documents/filters.py
@@ -114,6 +114,8 @@ def filter(self, qs, value):
         ctype = ContentType.objects.get_for_model(self.model)
         UserObjectPermission = get_user_obj_perms_model()
         GroupObjectPermission = get_group_obj_perms_model()
+        # see https://github.com/paperless-ngx/paperless-ngx/issues/5392, we limit subqueries
+        # to 1 because Postgres doesn't like returning > 1 row, but all we care about is > 0
         return (
             qs.filter(
                 owner_id=value,
@@ -123,7 +125,7 @@ def filter(self, qs, value):
                     UserObjectPermission.objects.filter(
                         content_type=ctype,
                         object_pk=Cast(OuterRef("pk"), CharField()),
-                    ).values("user_id"),
+                    ).values("user_id")[:1],
                 ),
             )
             .annotate(
@@ -131,7 +133,7 @@ def filter(self, qs, value):
                     GroupObjectPermission.objects.filter(
                         content_type=ctype,
                         object_pk=Cast(OuterRef("pk"), CharField()),
-                    ).values("group_id"),
+                    ).values("group_id")[:1],
                 ),
             )
             .filter(


Test patch: