Repository: paperless-ngx/paperless-ngx

Issue:
[BUG] Autocompletion suggests terms included in documents a user has no access to
### Description

When using the autocomplete feature from `/api/search/autocomplete/?term=...`, the API suggests terms included in documents which the user has no access to (in 1.14.0).

This has one "unfortunate" side effect and another more severe effect:

1. A user without access to the document the suggestion string originated from will be suggested the string but will not get any results. 
2. The more severe case: Imagine someone uploads a very sensitive document, e.g. containing your social security number. In some countries, these numbers follow a certain format, e.g. start with a constant expression (like DE123). Now, if a user without access to this document wants to know the social security number of another user (assuming a corresponding document with this number exists), he can just search for "DE123" and be suggested the whole social security number. For instance, in germany, the payroll you receive at the end of each month contains your tax id. Not sure I want to share this with other users.

So one could essentially do "text mining" of documents one should not have access to.

I'm not sure how the search is currently implemented, but I assume it is a global index which is built from the content of _all_ documents. If this is the case, this would be a lot of effort to fix since we would have to
1. Create this index for each user independently based on the documents this user has access to, and
2. Update the index every time the permissions for this user change or someone shares/removes a document from a user.

### Steps to reproduce

1. Create PDF file containing only one unique String (like 123456789)
2. Upload PDF to paperless as admin (or any other user with add document permission)
3. Assign the uploader (here: admin) as the owner and add per document permissions to admin (i.e. no other user is allowed to view the document)
4. Logout
5. Login as user B and ensure this user has the **document.view** permission
6. Ensure that user B does not see the new document (no access!)
7.  Search for the unique string in the top
8. Observe that it is auto completed

### Webserver logs

```bash
No related logs.
```


### Browser logs

```bash
No related logs
```


### Paperless-ngx version

1.14.0

### Host OS

Ubuntu 20.04

### Installation method

Docker - official image

### Browser

Chrome

### Configuration changes

_No response_

### Other

_No response_


Discussion:
Autocomplete uses https://whoosh.readthedocs.io/en/latest/api/reading.html#whoosh.reading.IndexReader.most_distinctive_terms , I don't see an obvious solution here
I see. After reading the docs, I think the only solution would be to maintain several indices which whould also have to be updated on permission change, so that each index is built based solely on the documents a user also has access to. Not sure how big these indices are, but if there are many users with mostly shared documents, the overall space required for these indices would increase quite drastically.
I agree, should be fairly complex to implement as far as I can tell. 
 
Would it help to add the permission relevant fields to the index, like owner etc.
And then check all the entries after searching for permission?
It already is there. The issue is the method used to generate these, noted above, doesn't include any kind of filtering of documents.
Sorry I'm not familiar with the code right now.

But couldn't we just copy the method to the paperless code and add filtering or write a custom query?
> Sorry I'm not familiar with the code right now.
> 
> But couldn't we just copy the method to the paperless code and add filtering or write a custom query?

If you look into the code a bit I think you will see it's not quite so simple. Would welcome a PR, of course.
That's what I came up with in the short time.
Because I'm no python dev I not really able to submit a PR.  That's why I did it in a test case

But at least this seams to work in the same way as the original method.
Return values and score are the same.

Still needs the permission filtering in the query and sure some python improvements.

```python
   def test_auto_complete_custom(self):
        doc1 = Document.objects.create(
            title="doc1",
            checksum="A",
            content="test test2 test3",
        )
        doc2 = Document.objects.create(title="doc2", checksum="B", content="test test2")
        doc3 = Document.objects.create(title="doc3", checksum="C", content="test2")

        index.add_or_update_document(doc1)
        index.add_or_update_document(doc2)
        index.add_or_update_document(doc3)

        ix = index.open_index()

        results = self.autocomplete(ix)

        self.assertListEqual(
            results,
            [b"test3", b"test", b"test2"],
        )

    def autocomplete(self, ix):
        with ix.searcher(weighting=scoring.TF_IDF()) as s:
            qp = QueryParser("content", schema=ix.schema)
            q = qp.parse("tes*")

            results = s.search(q, terms=True)

            termCount = dict()
            if results.has_matched_terms():
                for hit in results:
                    for term in hit.matched_terms():
                        if term[1] in termCount:
                            termCount[term[1]] = termCount[term[1]] + 1
                        else:
                            termCount[term[1]] = 1


            scoreMap = OrderedDict()
            N = float(ix.reader().doc_count())
            for key in termCount:
                scoreMap[key] = (termCount[key] * log(N / termCount[key]))

            return list(scoreMap.keys())
```


> That's what I came up with in the short time. Because I'm no python dev I not really able to submit a PR. That's why I did it in a test case
> 
> But at least this seams to work in the same way as the original method. Return values and score are the same.

Thanks for the snippet :) It’s not really the logic that concerns me it’s the performance of eg this vs how autocomplete is “natively” implemented, but to be honest I haven’t tried yet so you’re inspiring me to at least see how it goes! Either way, thanks for the input. I’ll report back once I have some time to tinker

Resolving patch:
diff --git a/src/documents/index.py b/src/documents/index.py
--- a/src/documents/index.py
+++ b/src/documents/index.py
@@ -1,6 +1,7 @@
 import logging
 import math
 import os
+from collections import Counter
 from contextlib import contextmanager
 
 from dateutil.parser import isoparse
@@ -17,17 +18,21 @@
 from whoosh.fields import TEXT
 from whoosh.fields import Schema
 from whoosh.highlight import HtmlFormatter
+from whoosh.index import FileIndex
 from whoosh.index import create_in
 from whoosh.index import exists_in
 from whoosh.index import open_dir
 from whoosh.qparser import MultifieldParser
+from whoosh.qparser import QueryParser
 from whoosh.qparser.dateparse import DateParserPlugin
+from whoosh.scoring import TF_IDF
 from whoosh.searching import ResultsPage
 from whoosh.searching import Searcher
 from whoosh.writing import AsyncWriter
 
 from documents.models import Document
 from documents.models import Note
+from documents.models import User
 
 logger = logging.getLogger("paperless.index")
 
@@ -238,15 +243,9 @@ def _get_query_filter(self):
             elif k == "storage_path__isnull":
                 criterias.append(query.Term("has_path", v == "false"))
 
-        user_criterias = [query.Term("has_owner", False)]
-        if "user" in self.query_params:
-            if self.query_params["is_superuser"]:  # superusers see all docs
-                user_criterias = []
-            else:
-                user_criterias.append(query.Term("owner_id", self.query_params["user"]))
-                user_criterias.append(
-                    query.Term("viewer_id", str(self.query_params["user"])),
-                )
+        user_criterias = get_permissions_criterias(
+            user=self.user,
+        )
         if len(criterias) > 0:
             if len(user_criterias) > 0:
                 criterias.append(query.Or(user_criterias))
@@ -282,12 +281,13 @@ def _get_query_sortedby(self):
         else:
             return sort_fields_map[field], reverse
 
-    def __init__(self, searcher: Searcher, query_params, page_size):
+    def __init__(self, searcher: Searcher, query_params, page_size, user):
         self.searcher = searcher
         self.query_params = query_params
         self.page_size = page_size
         self.saved_results = dict()
         self.first_score = None
+        self.user = user
 
     def __len__(self):
         page = self[0:1]
@@ -368,13 +368,42 @@ def _get_query(self):
         return q, mask
 
 
-def autocomplete(ix, term, limit=10):
-    with ix.reader() as reader:
-        terms = []
-        for score, t in reader.most_distinctive_terms(
-            "content",
-            number=limit,
-            prefix=term.lower(),
-        ):
-            terms.append(t)
-        return terms
+def autocomplete(ix: FileIndex, term: str, limit: int = 10, user: User = None):
+    """
+    Mimics whoosh.reading.IndexReader.most_distinctive_terms with permissions
+    and without scoring
+    """
+    terms = []
+
+    with ix.searcher(weighting=TF_IDF()) as s:
+        qp = QueryParser("content", schema=ix.schema)
+        q = qp.parse(f"{term.lower()}*")
+        user_criterias = get_permissions_criterias(user)
+
+        results = s.search(
+            q,
+            terms=True,
+            filter=query.Or(user_criterias) if user_criterias is not None else None,
+        )
+
+        termCounts = Counter()
+        if results.has_matched_terms():
+            for hit in results:
+                for _, term in hit.matched_terms():
+                    termCounts[term] += 1
+            terms = [t for t, _ in termCounts.most_common(limit)]
+
+    return terms
+
+
+def get_permissions_criterias(user: User = None):
+    user_criterias = [query.Term("has_owner", False)]
+    if user is not None:
+        if user.is_superuser:  # superusers see all docs
+            user_criterias = []
+        else:
+            user_criterias.append(query.Term("owner_id", user.id))
+            user_criterias.append(
+                query.Term("viewer_id", str(user.id)),
+            )
+    return user_criterias
diff --git a/src/documents/views.py b/src/documents/views.py
--- a/src/documents/views.py
+++ b/src/documents/views.py
@@ -598,15 +598,6 @@ def filter_queryset(self, queryset):
         if self._is_search_request():
             from documents import index
 
-            if hasattr(self.request, "user"):
-                # pass user to query for perms
-                self.request.query_params._mutable = True
-                self.request.query_params["user"] = self.request.user.id
-                self.request.query_params[
-                    "is_superuser"
-                ] = self.request.user.is_superuser
-                self.request.query_params._mutable = False
-
             if "query" in self.request.query_params:
                 query_class = index.DelayedFullTextQuery
             elif "more_like_id" in self.request.query_params:
@@ -618,6 +609,7 @@ def filter_queryset(self, queryset):
                 self.searcher,
                 self.request.query_params,
                 self.paginator.get_page_size(self.request),
+                self.request.user,
             )
         else:
             return super().filter_queryset(queryset)
@@ -817,6 +809,8 @@ class SearchAutoCompleteView(APIView):
     permission_classes = (IsAuthenticated,)
 
     def get(self, request, format=None):
+        user = self.request.user if hasattr(self.request, "user") else None
+
         if "term" in request.query_params:
             term = request.query_params["term"]
         else:
@@ -833,7 +827,14 @@ def get(self, request, format=None):
 
         ix = index.open_index()
 
-        return Response(index.autocomplete(ix, term, limit))
+        return Response(
+            index.autocomplete(
+                ix,
+                term,
+                limit,
+                user,
+            ),
+        )
 
 
 class StatisticsView(APIView):


Test patch:
diff --git a/src/documents/tests/test_api.py b/src/documents/tests/test_api.py
--- a/src/documents/tests/test_api.py
+++ b/src/documents/tests/test_api.py
@@ -832,7 +832,7 @@ def test_search_added_in_last_month_timezone_behind(self):
 
     @mock.patch("documents.index.autocomplete")
     def test_search_autocomplete(self, m):
-        m.side_effect = lambda ix, term, limit: [term for _ in range(limit)]
+        m.side_effect = lambda ix, term, limit, user: [term for _ in range(limit)]
 
         response = self.client.get("/api/search/autocomplete/?term=test")
         self.assertEqual(response.status_code, status.HTTP_200_OK)
@@ -852,6 +852,66 @@ def test_search_autocomplete(self, m):
         self.assertEqual(response.status_code, status.HTTP_200_OK)
         self.assertEqual(len(response.data), 10)
 
+    def test_search_autocomplete_respect_permissions(self):
+        """
+        GIVEN:
+            - Multiple users and documents with & without permissions
+        WHEN:
+            - API reuqest for autocomplete is made by user with or without permissions
+        THEN:
+            - Terms only within docs user has access to are returned
+        """
+        u1 = User.objects.create_user("user1")
+        u2 = User.objects.create_user("user2")
+
+        self.client.force_authenticate(user=u1)
+
+        d1 = Document.objects.create(
+            title="doc1",
+            content="apples",
+            checksum="1",
+            owner=u1,
+        )
+        d2 = Document.objects.create(
+            title="doc2",
+            content="applebaum",
+            checksum="2",
+            owner=u1,
+        )
+        d3 = Document.objects.create(
+            title="doc3",
+            content="appletini",
+            checksum="3",
+            owner=u1,
+        )
+
+        with AsyncWriter(index.open_index()) as writer:
+            index.update_document(writer, d1)
+            index.update_document(writer, d2)
+            index.update_document(writer, d3)
+
+        response = self.client.get("/api/search/autocomplete/?term=app")
+        self.assertEqual(response.status_code, status.HTTP_200_OK)
+        self.assertEqual(response.data, [b"apples", b"applebaum", b"appletini"])
+
+        d3.owner = u2
+
+        with AsyncWriter(index.open_index()) as writer:
+            index.update_document(writer, d3)
+
+        response = self.client.get("/api/search/autocomplete/?term=app")
+        self.assertEqual(response.status_code, status.HTTP_200_OK)
+        self.assertEqual(response.data, [b"apples", b"applebaum"])
+
+        assign_perm("view_document", u1, d3)
+
+        with AsyncWriter(index.open_index()) as writer:
+            index.update_document(writer, d3)
+
+        response = self.client.get("/api/search/autocomplete/?term=app")
+        self.assertEqual(response.status_code, status.HTTP_200_OK)
+        self.assertEqual(response.data, [b"apples", b"applebaum", b"appletini"])
+
     @pytest.mark.skip(reason="Not implemented yet")
     def test_search_spelling_correction(self):
         with AsyncWriter(index.open_index()) as writer:
diff --git a/src/documents/tests/test_index.py b/src/documents/tests/test_index.py
--- a/src/documents/tests/test_index.py
+++ b/src/documents/tests/test_index.py
@@ -25,13 +25,13 @@ def test_auto_complete(self):
 
         self.assertListEqual(
             index.autocomplete(ix, "tes"),
-            [b"test3", b"test", b"test2"],
+            [b"test2", b"test", b"test3"],
         )
         self.assertListEqual(
             index.autocomplete(ix, "tes", limit=3),
-            [b"test3", b"test", b"test2"],
+            [b"test2", b"test", b"test3"],
         )
-        self.assertListEqual(index.autocomplete(ix, "tes", limit=1), [b"test3"])
+        self.assertListEqual(index.autocomplete(ix, "tes", limit=1), [b"test2"])
         self.assertListEqual(index.autocomplete(ix, "tes", limit=0), [])
 
     def test_archive_serial_number_ranging(self):