Repository: paperless-ngx/paperless-ngx

Issue:
[BUG] Re-indexing fails with out of range error
### Description

Since 1.12.2, the docker container of Paperless crashes when trying to rebuild the search index.

### Steps to reproduce

I don't know.

### Webserver logs

```bash
paperless  | Waiting for Redis...
paperless  | Connected to Redis broker.
paperless  | Apply database migrations...
paperless  | Operations to perform:
paperless  |   Apply all migrations: admin, auth, authtoken, contenttypes, django_celery_results, documents, paperless_mail, sessions
paperless  | Running migrations:
paperless  |   No migrations to apply.
paperless  | Running Django checks
paperless  | System check identified no issues (0 silenced).
paperless  | Search index out of date. Updating...
paperless  | Traceback (most recent call last):
paperless  |   File "/usr/src/paperless/src/manage.py", line 11, in <module>
paperless  |     execute_from_command_line(sys.argv)
paperless  |   File "/usr/local/lib/python3.9/site-packages/django/core/management/__init__.py", line 446, in execute_from_command_line
paperless  |     utility.execute()
paperless  |   File "/usr/local/lib/python3.9/site-packages/django/core/management/__init__.py", line 440, in execute
paperless  |     self.fetch_command(subcommand).run_from_argv(self.argv)
paperless  |   File "/usr/local/lib/python3.9/site-packages/django/core/management/base.py", line 402, in run_from_argv
paperless  |     self.execute(*args, **cmd_options)
paperless  |   File "/usr/local/lib/python3.9/site-packages/django/core/management/base.py", line 448, in execute
paperless  |     output = self.handle(*args, **options)
paperless  |   File "/usr/src/paperless/src/documents/management/commands/document_index.py", line 23, in handle
paperless  |     index_reindex(progress_bar_disable=options["no_progress_bar"])
paperless  |   File "/usr/src/paperless/src/documents/tasks.py", line 56, in index_reindex
paperless  |     index.update_document(writer, document)
paperless  |   File "/usr/src/paperless/src/documents/index.py", line 96, in update_document
paperless  |     writer.update_document(
paperless  |   File "/usr/local/lib/python3.9/site-packages/whoosh/writing.py", line 1024, in update_document
paperless  |     self._record("update_document", args, kwargs)
paperless  |   File "/usr/local/lib/python3.9/site-packages/whoosh/writing.py", line 1001, in _record
paperless  |     getattr(self.writer, method)(*args, **kwargs)
paperless  |   File "/usr/local/lib/python3.9/site-packages/whoosh/writing.py", line 490, in update_document
paperless  |     self.add_document(**fields)
paperless  |   File "/usr/local/lib/python3.9/site-packages/whoosh/writing.py", line 750, in add_document
paperless  |     for tbytes, freq, weight, vbytes in items:
paperless  |   File "/usr/local/lib/python3.9/site-packages/whoosh/fields.py", line 669, in index
paperless  |     yield (self.to_bytes(num, shift), 1, 1.0, emptybytes)
paperless  |   File "/usr/local/lib/python3.9/site-packages/whoosh/fields.py", line 722, in to_bytes
paperless  |     x = self.prepare_number(x)
paperless  |   File "/usr/local/lib/python3.9/site-packages/whoosh/fields.py", line 691, in prepare_number
paperless  |     raise ValueError("Numeric field value %s out of range [%s, %s]"
paperless  | ValueError: Numeric field value 1460318454054548 out of range [0, 4294967295]
```


### Browser logs

_No response_

### Paperless-ngx version

1.12.2

### Host OS

Archlinux

### Installation method

Docker - official image

### Browser

_No response_

### Configuration changes

_No response_

### Other

_No response_


Discussion:
I'm guessing you have an ASN with a value 1460318454054548.  That is indeed outside the supported range.  The database will no longer allow those either.
I don't know what an ASN is and I haven't entered any manually AFAIK. The error doesn't give me precise info about the offending document.

Either way, it's in an existing document and I don't think a value not being allowed anymore should make an update fail and the app completely stop working. Maybe it could notify the admin or show an error on the dashboard. 

Luckily, I can revert the container to 1.12.1. But what if I couldn't? The re-indexing happens after the database migrations, and if those migrations aren't backward-compatible, you can't go back to your previous version unless you have a proper backup.
So v1.12.1 works if you run the `reindex` command?  That's odd as the index itself didn't change between those versions.

It was a bug that a value over 1 quadrillion was ever accepted and merely incidental the database happened to be able to store it.  I suppose it's always [some one's workflow](https://xkcd.com/1172/).  The admin page for documents will let you have a look and filter by the ASN, to confirm it is that field.
> So v1.12.1 works if you run the reindex command? That's odd as the index itself didn't change between those versions.

Just tested: re-indexing doesn't work in v1.12.1 either. It didn't force a re-index at start-up though, so the container was able to start without crashing.

>  The admin page for documents will let you have a look and filter by the ASN, to confirm it is that field.

Thank you! I found the document and fixed it. v1.12.2 now starts up correctly.

Resolving patch:
diff --git a/src/documents/consumer.py b/src/documents/consumer.py
--- a/src/documents/consumer.py
+++ b/src/documents/consumer.py
@@ -146,11 +146,16 @@ def pre_check_asn_value(self):
             return
         # Validate the range is above zero and less than uint32_t max
         # otherwise, Whoosh can't handle it in the index
-        if self.override_asn < 0 or self.override_asn > 0xFF_FF_FF_FF:
+        if (
+            self.override_asn < Document.ARCHIVE_SERIAL_NUMBER_MIN
+            or self.override_asn > Document.ARCHIVE_SERIAL_NUMBER_MAX
+        ):
             self._fail(
                 MESSAGE_ASN_RANGE,
                 f"Not consuming {self.filename}: "
-                f"Given ASN {self.override_asn} is out of range [0, 4,294,967,295]",
+                f"Given ASN {self.override_asn} is out of range "
+                f"[{Document.ARCHIVE_SERIAL_NUMBER_MIN:,}, "
+                f"{Document.ARCHIVE_SERIAL_NUMBER_MAX:,}]",
             )
         if Document.objects.filter(archive_serial_number=self.override_asn).exists():
             self._fail(
diff --git a/src/documents/index.py b/src/documents/index.py
--- a/src/documents/index.py
+++ b/src/documents/index.py
@@ -90,10 +90,22 @@ def open_index_searcher():
         searcher.close()
 
 
-def update_document(writer, doc):
+def update_document(writer: AsyncWriter, doc: Document):
     tags = ",".join([t.name for t in doc.tags.all()])
     tags_ids = ",".join([str(t.id) for t in doc.tags.all()])
     comments = ",".join([str(c.comment) for c in Comment.objects.filter(document=doc)])
+    asn = doc.archive_serial_number
+    if asn is not None and (
+        asn < Document.ARCHIVE_SERIAL_NUMBER_MIN
+        or asn > Document.ARCHIVE_SERIAL_NUMBER_MAX
+    ):
+        logger.error(
+            f"Not indexing Archive Serial Number {asn} of document {doc.pk}. "
+            f"ASN is out of range "
+            f"[{Document.ARCHIVE_SERIAL_NUMBER_MIN:,}, "
+            f"{Document.ARCHIVE_SERIAL_NUMBER_MAX:,}.",
+        )
+        asn = 0
     writer.update_document(
         id=doc.pk,
         title=doc.title,
@@ -109,7 +121,7 @@ def update_document(writer, doc):
         has_type=doc.document_type is not None,
         created=doc.created,
         added=doc.added,
-        asn=doc.archive_serial_number,
+        asn=asn,
         modified=doc.modified,
         path=doc.storage_path.name if doc.storage_path else None,
         path_id=doc.storage_path.id if doc.storage_path else None,
diff --git a/src/documents/models.py b/src/documents/models.py
--- a/src/documents/models.py
+++ b/src/documents/models.py
@@ -3,6 +3,7 @@
 import os
 import re
 from collections import OrderedDict
+from typing import Final
 from typing import Optional
 
 import dateutil.parser
@@ -229,6 +230,9 @@ class Document(models.Model):
         help_text=_("The original name of the file when it was uploaded"),
     )
 
+    ARCHIVE_SERIAL_NUMBER_MIN: Final[int] = 0
+    ARCHIVE_SERIAL_NUMBER_MAX: Final[int] = 0xFF_FF_FF_FF
+
     archive_serial_number = models.PositiveIntegerField(
         _("archive serial number"),
         blank=True,
@@ -236,8 +240,8 @@ class Document(models.Model):
         unique=True,
         db_index=True,
         validators=[
-            MaxValueValidator(0xFF_FF_FF_FF),
-            MinValueValidator(0),
+            MaxValueValidator(ARCHIVE_SERIAL_NUMBER_MAX),
+            MinValueValidator(ARCHIVE_SERIAL_NUMBER_MIN),
         ],
         help_text=_(
             "The position of this document in your physical document " "archive.",


Test patch:
diff --git a/src/documents/tests/test_index.py b/src/documents/tests/test_index.py
--- a/src/documents/tests/test_index.py
+++ b/src/documents/tests/test_index.py
@@ -1,3 +1,5 @@
+from unittest import mock
+
 from django.test import TestCase
 from documents import index
 from documents.models import Document
@@ -31,3 +33,60 @@ def test_auto_complete(self):
         )
         self.assertListEqual(index.autocomplete(ix, "tes", limit=1), [b"test3"])
         self.assertListEqual(index.autocomplete(ix, "tes", limit=0), [])
+
+    def test_archive_serial_number_ranging(self):
+        """
+        GIVEN:
+            - Document with an archive serial number above schema allowed size
+        WHEN:
+            - Document is provided to the index
+        THEN:
+            - Error is logged
+            - Document ASN is reset to 0 for the index
+        """
+        doc1 = Document.objects.create(
+            title="doc1",
+            checksum="A",
+            content="test test2 test3",
+            # yes, this is allowed, unless full_clean is run
+            # DRF does call the validators, this test won't
+            archive_serial_number=Document.ARCHIVE_SERIAL_NUMBER_MAX + 1,
+        )
+        with self.assertLogs("paperless.index", level="ERROR") as cm:
+            with mock.patch(
+                "documents.index.AsyncWriter.update_document",
+            ) as mocked_update_doc:
+                index.add_or_update_document(doc1)
+
+                mocked_update_doc.assert_called_once()
+                _, kwargs = mocked_update_doc.call_args
+
+                self.assertEqual(kwargs["asn"], 0)
+
+                error_str = cm.output[0]
+                expected_str = "ERROR:paperless.index:Not indexing Archive Serial Number 4294967296 of document 1"
+                self.assertIn(expected_str, error_str)
+
+    def test_archive_serial_number_is_none(self):
+        """
+        GIVEN:
+            - Document with no archive serial number
+        WHEN:
+            - Document is provided to the index
+        THEN:
+            - ASN isn't touched
+        """
+        doc1 = Document.objects.create(
+            title="doc1",
+            checksum="A",
+            content="test test2 test3",
+        )
+        with mock.patch(
+            "documents.index.AsyncWriter.update_document",
+        ) as mocked_update_doc:
+            index.add_or_update_document(doc1)
+
+            mocked_update_doc.assert_called_once()
+            _, kwargs = mocked_update_doc.call_args
+
+            self.assertIsNone(kwargs["asn"])