Repository: paperless-ngx/paperless-ngx

Issue:
[BUG] data loss in import: temporary failure in name resolution
### Description

import on a very busy server fails with "temporary failure in name resolution"

This means the API result was "OK", nevertheless import fails, leading to data loss as the uploading client will already have deleted the file but it is nowhere to be found on the server. To add insult to injury, there does not seem to be a way to retry the import even though the very nature of the error is temporary. 

So, this should never happen for two reasons:

1. if the API result for an import is "OK" then the document absolutely must be imported
2. if there is a temporary error condition for whatever reason, it should not lead to a permanent failure.



### Steps to reproduce

1. import document using the API
2. on a very busy server observe the failed import and "temporary failure in name resolution" in webserver logs.

### Webserver logs

```bash
20:05:31 [Q] INFO Process-1:598 ready for work at 2921
20:05:31 [Q] INFO Process-1:598 processing [scan2paperless_10471.pdf]
20:05:31 [Q] ERROR Failed [scan2paperless_9987.pdf] - [Errno -3] Temporary failure in name resolution : Traceback (most recent call last):
  File "/usr/src/paperless/src/docker/src/django-q/django_q/cluster.py", line 434, in worker
    res = f(*task["args"], **task["kwargs"])
  File "/usr/src/paperless/src/documents/tasks.py", line 162, in consume_file
    document = Consumer().try_consume_file(
  File "/usr/src/paperless/src/documents/consumer.py", line 314, in try_consume_file
    self._send_progress(70, 100, "WORKING", MESSAGE_GENERATING_THUMBNAIL)
  File "/usr/src/paperless/src/documents/consumer.py", line 76, in _send_progress
    async_to_sync(self.channel_layer.group_send)(
  File "/usr/local/lib/python3.9/site-packages/asgiref/sync.py", line 218, in __call__
    return call_result.result()
  File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 439, in result
    return self.__get_result()
  File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.9/site-packages/asgiref/sync.py", line 284, in main_wrap
    result = await self.awaitable(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/channels_redis/core.py", line 666, in group_send
    async with self.connection(self.consistent_hash(group)) as connection:
  File "/usr/local/lib/python3.9/site-packages/channels_redis/core.py", line 862, in __aenter__
    self.conn = await self.pool.pop()
  File "/usr/local/lib/python3.9/site-packages/channels_redis/core.py", line 93, in pop
    conn = await self.create_conn(loop)
  File "/usr/local/lib/python3.9/site-packages/channels_redis/core.py", line 79, in create_conn
    return await aioredis.create_redis_pool(**kwargs)
  File "/usr/local/lib/python3.9/site-packages/aioredis/commands/__init__.py", line 188, in create_redis_pool
    pool = await create_pool(address, db=db,
  File "/usr/local/lib/python3.9/site-packages/aioredis/pool.py", line 58, in create_pool
    await pool._fill_free(override_min=False)
  File "/usr/local/lib/python3.9/site-packages/aioredis/pool.py", line 383, in _fill_free
    conn = await self._create_new_connection(self._address)
  File "/usr/local/lib/python3.9/site-packages/aioredis/connection.py", line 111, in create_connection
    reader, writer = await asyncio.wait_for(open_connection(
  File "/usr/local/lib/python3.9/asyncio/tasks.py", line 442, in wait_for
    return await fut
  File "/usr/local/lib/python3.9/site-packages/aioredis/stream.py", line 23, in open_connection
    transport, _ = await get_event_loop().create_connection(
  File "/usr/local/lib/python3.9/asyncio/base_events.py", line 1026, in create_connection
    infos = await self._ensure_resolved(
  File "/usr/local/lib/python3.9/asyncio/base_events.py", line 1405, in _ensure_resolved
    return await loop.getaddrinfo(host, port, family=family, type=type,
  File "/usr/local/lib/python3.9/asyncio/base_events.py", line 861, in getaddrinfo
    return await self.run_in_executor(
  File "/usr/local/lib/python3.9/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.9/socket.py", line 954, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Temporary failure in name resolution
```


### Browser logs

_No response_

### Paperless-ngx version

1.8.0

### Host OS

debian / x86-64

### Installation method

Docker - official image

### Browser

firefox

### Configuration changes

_No response_

### Other

_No response_


Discussion:
For starters, I would highly suggest upgrading to at least 1.10.x.  `django-q` is no longer utilized and hasn't seen updates in quite some time.  In particular, `redis-py` has been updated, which is where the traceback leads.

Uploads and consumption are asynchronous.  That's very baked into the design and not going to change.  All the OK response means is the file was received without error.

As to the DNS problem, that sounds like something with the Docker networking when attempting to talk with the broker.  I don't think the busyness of the server should affect it in anyway.
One of the side-effects of the current design is:

1. program on machine running api client scans a document into a file
2. api client uploads the file into paperless
3. api client receives "OK"
4. api client deletes the file
5. user of the api client discards the document
6. paperless does not import the file
7. data is missing

This is a problem. Either clients must only receive "OK" when the document is ingested - you have made the point that the design is to not do this (and there are good reasons not to). Or they must have a way to check for the success of the consumption process (which they, according to docs don't, as no other information is passed to the client which they could use to query the search api).

Apart from the API design topic, there is still a bug in handling a temporary error causing permanent failure.


From reading the changelog it does not look like an upgrade will help with this particular problem.

That being said, of course an upgrade will be done.

Edit: Update was done to 1.11.1
It looks simple enough to return the UUID of the created task, which can then be queried for status against the tasks API endpoint.  That response includes status, as well as some filenames and timing information.

I made it a query, so the URL looks like http://localhost:8180/api/tasks/?task_id=69942b2b-beba-429b-bde3-25bd9da7cf8a

Resolving patch:
diff --git a/src/documents/views.py b/src/documents/views.py
--- a/src/documents/views.py
+++ b/src/documents/views.py
@@ -617,7 +617,7 @@ def post(self, request, *args, **kwargs):
 
         task_id = str(uuid.uuid4())
 
-        consume_file.delay(
+        async_task = consume_file.delay(
             temp_filename,
             override_filename=doc_name,
             override_title=title,
@@ -628,7 +628,7 @@ def post(self, request, *args, **kwargs):
             override_created=created,
         )
 
-        return Response("OK")
+        return Response(async_task.id)
 
 
 class SelectionDataView(GenericAPIView):
@@ -886,13 +886,18 @@ class TasksViewSet(ReadOnlyModelViewSet):
     permission_classes = (IsAuthenticated,)
     serializer_class = TasksViewSerializer
 
-    queryset = (
-        PaperlessTask.objects.filter(
-            acknowledged=False,
+    def get_queryset(self):
+        queryset = (
+            PaperlessTask.objects.filter(
+                acknowledged=False,
+            )
+            .order_by("date_created")
+            .reverse()
         )
-        .order_by("date_created")
-        .reverse()
-    )
+        task_id = self.request.query_params.get("task_id")
+        if task_id is not None:
+            queryset = PaperlessTask.objects.filter(task_id=task_id)
+        return queryset
 
 
 class AcknowledgeTasksView(GenericAPIView):


Test patch:
diff --git a/src/documents/tests/test_api.py b/src/documents/tests/test_api.py
--- a/src/documents/tests/test_api.py
+++ b/src/documents/tests/test_api.py
@@ -793,6 +793,8 @@ def test_statistics_no_inbox_tag(self):
     @mock.patch("documents.views.consume_file.delay")
     def test_upload(self, m):
 
+        m.return_value = celery.result.AsyncResult(id=str(uuid.uuid4()))
+
         with open(
             os.path.join(os.path.dirname(__file__), "samples", "simple.pdf"),
             "rb",
@@ -816,6 +818,8 @@ def test_upload(self, m):
     @mock.patch("documents.views.consume_file.delay")
     def test_upload_empty_metadata(self, m):
 
+        m.return_value = celery.result.AsyncResult(id=str(uuid.uuid4()))
+
         with open(
             os.path.join(os.path.dirname(__file__), "samples", "simple.pdf"),
             "rb",
@@ -839,6 +843,8 @@ def test_upload_empty_metadata(self, m):
     @mock.patch("documents.views.consume_file.delay")
     def test_upload_invalid_form(self, m):
 
+        m.return_value = celery.result.AsyncResult(id=str(uuid.uuid4()))
+
         with open(
             os.path.join(os.path.dirname(__file__), "samples", "simple.pdf"),
             "rb",
@@ -853,6 +859,8 @@ def test_upload_invalid_form(self, m):
     @mock.patch("documents.views.consume_file.delay")
     def test_upload_invalid_file(self, m):
 
+        m.return_value = celery.result.AsyncResult(id=str(uuid.uuid4()))
+
         with open(
             os.path.join(os.path.dirname(__file__), "samples", "simple.zip"),
             "rb",
@@ -866,6 +874,9 @@ def test_upload_invalid_file(self, m):
 
     @mock.patch("documents.views.consume_file.delay")
     def test_upload_with_title(self, async_task):
+
+        async_task.return_value = celery.result.AsyncResult(id=str(uuid.uuid4()))
+
         with open(
             os.path.join(os.path.dirname(__file__), "samples", "simple.pdf"),
             "rb",
@@ -884,6 +895,9 @@ def test_upload_with_title(self, async_task):
 
     @mock.patch("documents.views.consume_file.delay")
     def test_upload_with_correspondent(self, async_task):
+
+        async_task.return_value = celery.result.AsyncResult(id=str(uuid.uuid4()))
+
         c = Correspondent.objects.create(name="test-corres")
         with open(
             os.path.join(os.path.dirname(__file__), "samples", "simple.pdf"),
@@ -903,6 +917,9 @@ def test_upload_with_correspondent(self, async_task):
 
     @mock.patch("documents.views.consume_file.delay")
     def test_upload_with_invalid_correspondent(self, async_task):
+
+        async_task.return_value = celery.result.AsyncResult(id=str(uuid.uuid4()))
+
         with open(
             os.path.join(os.path.dirname(__file__), "samples", "simple.pdf"),
             "rb",
@@ -917,6 +934,9 @@ def test_upload_with_invalid_correspondent(self, async_task):
 
     @mock.patch("documents.views.consume_file.delay")
     def test_upload_with_document_type(self, async_task):
+
+        async_task.return_value = celery.result.AsyncResult(id=str(uuid.uuid4()))
+
         dt = DocumentType.objects.create(name="invoice")
         with open(
             os.path.join(os.path.dirname(__file__), "samples", "simple.pdf"),
@@ -936,6 +956,9 @@ def test_upload_with_document_type(self, async_task):
 
     @mock.patch("documents.views.consume_file.delay")
     def test_upload_with_invalid_document_type(self, async_task):
+
+        async_task.return_value = celery.result.AsyncResult(id=str(uuid.uuid4()))
+
         with open(
             os.path.join(os.path.dirname(__file__), "samples", "simple.pdf"),
             "rb",
@@ -950,6 +973,9 @@ def test_upload_with_invalid_document_type(self, async_task):
 
     @mock.patch("documents.views.consume_file.delay")
     def test_upload_with_tags(self, async_task):
+
+        async_task.return_value = celery.result.AsyncResult(id=str(uuid.uuid4()))
+
         t1 = Tag.objects.create(name="tag1")
         t2 = Tag.objects.create(name="tag2")
         with open(
@@ -970,6 +996,9 @@ def test_upload_with_tags(self, async_task):
 
     @mock.patch("documents.views.consume_file.delay")
     def test_upload_with_invalid_tags(self, async_task):
+
+        async_task.return_value = celery.result.AsyncResult(id=str(uuid.uuid4()))
+
         t1 = Tag.objects.create(name="tag1")
         t2 = Tag.objects.create(name="tag2")
         with open(
@@ -986,6 +1015,9 @@ def test_upload_with_invalid_tags(self, async_task):
 
     @mock.patch("documents.views.consume_file.delay")
     def test_upload_with_created(self, async_task):
+
+        async_task.return_value = celery.result.AsyncResult(id=str(uuid.uuid4()))
+
         created = datetime.datetime(
             2022,
             5,
@@ -2948,6 +2980,59 @@ def test_get_tasks(self):
         self.assertEqual(returned_task2["status"], celery.states.PENDING)
         self.assertEqual(returned_task2["task_file_name"], task2.task_file_name)
 
+    def test_get_single_task_status(self):
+        """
+        GIVEN
+            - Query parameter for a valid task ID
+        WHEN:
+            - API call is made to get task status
+        THEN:
+            - Single task data is returned
+        """
+
+        id1 = str(uuid.uuid4())
+        task1 = PaperlessTask.objects.create(
+            task_id=id1,
+            task_file_name="task_one.pdf",
+        )
+
+        _ = PaperlessTask.objects.create(
+            task_id=str(uuid.uuid4()),
+            task_file_name="task_two.pdf",
+        )
+
+        response = self.client.get(self.ENDPOINT + f"?task_id={id1}")
+
+        self.assertEqual(response.status_code, 200)
+        self.assertEqual(len(response.data), 1)
+        returned_task1 = response.data[0]
+
+        self.assertEqual(returned_task1["task_id"], task1.task_id)
+
+    def test_get_single_task_status_not_valid(self):
+        """
+        GIVEN
+            - Query parameter for a non-existent task ID
+        WHEN:
+            - API call is made to get task status
+        THEN:
+            - No task data is returned
+        """
+        task1 = PaperlessTask.objects.create(
+            task_id=str(uuid.uuid4()),
+            task_file_name="task_one.pdf",
+        )
+
+        _ = PaperlessTask.objects.create(
+            task_id=str(uuid.uuid4()),
+            task_file_name="task_two.pdf",
+        )
+
+        response = self.client.get(self.ENDPOINT + "?task_id=bad-task-id")
+
+        self.assertEqual(response.status_code, 200)
+        self.assertEqual(len(response.data), 0)
+
     def test_acknowledge_tasks(self):
         """
         GIVEN: