Repository: paperless-ngx/paperless-ngx

Issue:
[BUG] Regression in API create date
Hi all,

I like using paperless; it is a quite fine web application.

I recently updated to paperless-ngx 1.10.0-beta running in a docker container on a Raspi 4B (4 GB).

Besides uploading new files via the browser (works fine), I also use the REST API to provide new documents. Since the upgrade to 1.10.0-beta the REST API upload does not work anymore. The upload itself is successful, but when paperless tries to set the provided creation date, it fails:

```
[2022-11-22 12:20:32,436] [ERROR] [paperless.handlers] Creating PaperlessTask failed: malformed node or string: <ast.Call object at 0xb4ee8610>
[2022-11-22 12:20:32,573] [INFO] [paperless.consumer] Consuming PDFTest.pdf
[2022-11-22 12:20:32,577] [DEBUG] [paperless.consumer] Detected mime type: application/pdf
[2022-11-22 12:20:32,579] [INFO] [paperless.consumer] Executing pre-consume script /usr/src/paperless/script/preconsume-preconsumedoc.sh
[2022-11-22 12:20:32,607] [INFO] [paperless.consumer] /usr/src/paperless/script/preconsume-preconsumedoc.sh exited 0
[2022-11-22 12:20:32,612] [DEBUG] [paperless.consumer] Parser: RasterisedDocumentParser
[2022-11-22 12:20:32,635] [DEBUG] [paperless.consumer] Parsing PDFTest.pdf...
[2022-11-22 12:20:33,603] [DEBUG] [paperless.parsing.tesseract] Extracted text from PDF file /tmp/paperless/paperless-upload-gbq4ehve
[2022-11-22 12:20:33,846] [DEBUG] [paperless.parsing.tesseract] Calling OCRmyPDF with args: {'input_file': PosixPath('/tmp/paperless/paperless-upload-gbq4ehve'), 'output_file': '/tmp/paperless/paperless-acb916vd/archive.pdf', 'use_threads': True, 'jobs': 4, 'language': 'deu', 'output_type': 'pdfa', 'progress_bar': False, 'skip_text': True, 'clean': True, 'deskew': True, 'rotate_pages': True, 'rotate_pages_threshold': 12.0, 'sidecar': '/tmp/paperless/paperless-acb916vd/sidecar.txt'}
[2022-11-22 12:20:35,609] [DEBUG] [paperless.parsing.tesseract] Incomplete sidecar file: discarding.
[2022-11-22 12:20:36,337] [DEBUG] [paperless.parsing.tesseract] Extracted text from PDF file /tmp/paperless/paperless-acb916vd/archive.pdf
[2022-11-22 12:20:36,338] [DEBUG] [paperless.consumer] Generating thumbnail for PDFTest.pdf...
[2022-11-22 12:20:36,353] [DEBUG] [paperless.parsing] Execute: convert -density 300 -scale 500x5000> -alpha remove -strip -auto-orient /tmp/paperless/paperless-acb916vd/archive.pdf[0] /tmp/paperless/paperless-acb916vd/convert.webp
[2022-11-22 12:20:43,706] [DEBUG] [paperless.consumer] Saving record to database
[2022-11-22 12:20:43,707] [DEBUG] [paperless.consumer] Creation date from post_documents parameter: 2022-10-01T01:00:00+02:00
[2022-11-22 12:20:44,506] [ERROR] [paperless.consumer] The following error occurred while consuming PDFTest.pdf: 'str' object has no attribute 'utcoffset'
Traceback (most recent call last):
  File "/usr/src/paperless/src/documents/consumer.py", line 383, in try_consume_file
    document_consumption_finished.send(
  File "/usr/local/lib/python3.9/site-packages/django/dispatch/dispatcher.py", line 176, in send
    return [
  File "/usr/local/lib/python3.9/site-packages/django/dispatch/dispatcher.py", line 177, in <listcomp>
    (receiver, receiver(signal=self, sender=sender, **named))
  File "/usr/src/paperless/src/documents/signals/handlers.py", line 122, in set_document_type
    potential_document_type = matching.match_document_types(document, classifier)
  File "/usr/src/paperless/src/documents/matching.py", line 43, in match_document_types
    return list(
  File "/usr/src/paperless/src/documents/matching.py", line 44, in <lambda>
    filter(lambda o: matches(o, document) or o.pk == pred_id, document_types),
  File "/usr/src/paperless/src/documents/matching.py", line 104, in matches
    log_reason(matching_model, document, f"it contains this word: {word}")
  File "/usr/src/paperless/src/documents/matching.py", line 17, in log_reason
    f"{class_name} {matching_model.name} matched on document "
  File "/usr/src/paperless/src/documents/models.py", line 249, in __str__
    created = datetime.date.isoformat(timezone.localdate(self.created))
  File "/usr/local/lib/python3.9/site-packages/django/utils/timezone.py", line 232, in localdate
    return localtime(value, timezone).date()
  File "/usr/local/lib/python3.9/site-packages/django/utils/timezone.py", line 217, in localtime
    if is_naive(value):
  File "/usr/local/lib/python3.9/site-packages/django/utils/timezone.py", line 269, in is_naive
    return value.utcoffset() is None
AttributeError: 'str' object has no attribute 'utcoffset'
[2022-11-22 12:20:44,510] [DEBUG] [paperless.parsing.tesseract] Deleting directory /tmp/paperless/paperless-acb916vd
```

The upload is done via this (simplified) code:

```
#!/usr/bin/env python
import requests
import json

url = "http://<paperless-url>:<port>"
doc = "/path/to/a/pdf/file.pdf"

credentials = {"username": "<paperless-user>",
               "password": "<password>"}

sess = requests.Session ()

# ############## login
response = sess.get (url)
csrfToken = response.cookies ['csrftoken']
response = sess.post (url + "/api/token/",
                      data=credentials,
                      headers={'X-CSRFToken': csrfToken}
                     )

if response.status_code == 200:
  responseJSON = json.loads (response.text)
  authToken = responseJSON["token"]

  # ############## doc upload
  f = open (doc, "rb")
  filesPayload = {"document": f}
  otherPayload = {"correspondent": 20,
                  "title": "MyDocument",
                  "created": ("2022-10-01T00:00:00+01:00")
                  }
                  # also tried "created": ("2022-10-01 00:00Z")
  response = sess.post (url + "/api/documents/post_document/",
                        data=otherPayload,
                        files=filesPayload,
                        headers={"Authorization": f"Token {authToken}",
                                 f"X-CSRFToken": csrfToken}
                       )
  
  if response.status_code == 200:
    print ("successfully uploaded")
  else:
    print ("error while uploading")
    print (response.status_code)
    print (response.text)
```

When removing the "created" line from the "otherPayload" dictionary, everything works fine.

Is anyone able to reproduce this failure or is it just an error within my setup?

Regards,
Michael

_Originally posted by @kprnks in https://github.com/paperless-ngx/paperless-ngx/issues/1960#issuecomment-1323551285_
      


Discussion:
I can reproduce this between 1.10 and 1.9.  I'm not sure why, since that API endpoint didn't change.  It may be something in DRF.
~~I can’t dig into this right this second but is it related to #957 ?~~

edit: I guess timing doesn’t make sense that this would be the problem
Well, DRF is still giving us a `datetime.datetime`, as it should.  So somewhere, that gets `str'd`

Resolving patch:
diff --git a/src/documents/tasks.py b/src/documents/tasks.py
--- a/src/documents/tasks.py
+++ b/src/documents/tasks.py
@@ -3,6 +3,7 @@
 import os
 import shutil
 import uuid
+from datetime import datetime
 from pathlib import Path
 from typing import Type
 
@@ -98,6 +99,14 @@ def consume_file(
 
     path = Path(path).resolve()
 
+    # Celery converts this to a string, but everything expects a datetime
+    # Long term solution is to not use JSON for the serializer but pickle instead
+    if override_created is not None and isinstance(override_created, str):
+        try:
+            override_created = datetime.fromisoformat(override_created)
+        except Exception:
+            pass
+
     # check for separators in current document
     if settings.CONSUMER_ENABLE_BARCODES:
 


Test patch: