Repository: paperless-ngx/paperless-ngx

Issue:
[BUG] document_renamer dates in filename off by one day
### Description

After running the `document_renamer`, all dates in the filename are off by one day. E.g. a document with the date 11.05.2022 will have 10.05.2022 in the filename after running the exporter.

This is related to #267, however since the original issue was regarding the `document_exporter` which seems to be fixed in the new version v1.7.1, I decided to open a new issue.

### Steps to reproduce

1. (Since this is probably a time zone issue: Set Timezone to Europe/Berlin)
2. Set `PAPERLESS_FILENAME_FORMAT` to something that contains `{created_day}`
3. Modify some details of the document (e.g. the tile) using the web ui
4. The filename of the pdf will contain the correct document day
5. Now run the `document_renamer`
6. The document day in the file name will be off by one day

### Webserver logs

_No response_

### Paperless-ngx version

1.7.1

### Host OS

x86_64

### Installation method

Docker

### Browser

_No response_

### Configuration changes

_No response_

### Other

_No response_


Discussion:
If you use `{created}` as opposed to `{created_date}`, is the date correct then?
Nope, also `{created}` is off by one day.

Tested using the following filename format:
```
PAPERLESS_FILENAME_FORMAT={correspondent}/{created_year}/{document_type}/{created_year}{created_month}{created_day}_{created}_{correspondent}_{title}
```

Example result after running the `document_renamer` (document date is set to 14.02.2022):
![image](https://user-images.githubusercontent.com/39202639/168015092-43843c6f-5eea-45a7-a19f-b5a71b74f657.png)

After changing some doc properties using paperless it's correct again:
![image](https://user-images.githubusercontent.com/39202639/168015635-ff2cad2c-8ec5-4d9e-b4a2-894edcd03e52.png)

I haven't been able to reproduce this with 1.7.1.

I've got the settings:
```
TZ: "Europe/Berlin"
PAPERLESS_TIME_ZONE: "Europe/Berlin"
PAPERLESS_FILENAME_FORMAT: "{created_year}{created_month}{created_day}_{created}_{correspondent}_{title}"
```

Document created is 14.02.2022.  That produces (in documents/originals) a filename of `20220214_2022-02-14...`.  After running the renamer, the filename is unchanged.  On export, the filename is also corrected.  

I have also tried shutting down and changing the format, then doing a rename.  The filename was updated to the new format, but the date was still correct.

`date` inside the container returns CEST, so the timezone has definitely taken effect.
I didn't know of the `TZ`-Env-variable - it wasn't set and indeed running `date` inside the container the timezone had not taken effect. 
However, even after setting `TZ` to `Europe/Berlin` as well, the dates after running the `document_renamer` are still off by one day...
The `date` command returns the correct date now though...

Here's my full `docker-compose.env` (comments removed and some private details replaced with xxx):
```
USERMAP_UID=1026
USERMAP_GID=100

PAPERLESS_SECRET_KEY=xxx

PAPERLESS_TIME_ZONE=Europe/Berlin
TZ=Europe/Berlin

PAPERLESS_OCR_LANGUAGE=deu

PAPERLESS_TRASH_DIR=/usr/src/paperless/trash
PAPERLESS_FILENAME_FORMAT={correspondent}/{created_year}/{document_type}/{created_year}{created_month}{created_day}_{created}_{correspondent}_{title}
PAPERLESS_CONSUMER_DELETE_DUPLICATES=true

PAPERLESS_IGNORE_DATES=xx.xx.xxxx

PAPERLESS_URL=http://paperless

PAPERLESS_CONSUMER_ENABLE_BARCODES=true
```

Let me know if there's anything else you need me to check or send you to help you find the error.

Thanks.
TZ is a standard POSIX value (https://www.gnu.org/software/libc/manual/html_node/TZ-Variable.html), I tend to set it on all my containers, just is case.  In this case, I don't think it would matter, since django is using the PAPERLESS_TIME_ZONE as its setting.
Even matching to exact format, timezone, and language, I can't get this to happen.

1. Start from a clean slate
2. Start the containers
3. Upload a document
4. Change the created date via the web ui to 01/01/2022
5. document is stored as `media\documents\originals\Test\2022\empty\20220101_2022-01-01_Test_blank-page.pdf`
6. `docker exec -it paperless-ngx-dev-webserver /bin/bash`
7. `document_renamer`
8. document is still `20220101_2022-01-01_none_blank-page.pdf`
So now things are getting strange. I just setup a second instance of paperless to do some more testing. I uploaded only two documents, one test-document (simple MS Word export) and an actual scan (the one I used as an example before).

The test document works fine! The date in the filename stays correct after running the renamer. The scan however has the same wrong behaviour I described before...
The dest-document is fine in my main instance of paperless as well, so this issue seems to be document specific for some reason... Let me see if I can find a problematic document I can send you.
Okay, I think I'm one step closer. It seems to happen only with documents, were the date initially was automatically recognized. Please try with the following pdf:
[Test with date.pdf](https://github.com/paperless-ngx/paperless-ngx/files/8682045/Test.with.date.pdf)

Oh and by the way:
When running the `document_renamer` I always get the following warning. Don't know how I missed that before...
```
/usr/local/lib/python3.9/site-packages/dateparser/date_parser.py:35: PytzUsageWarning: The localize method is no longer necessary, as this time zone supports the fold attribute (PEP 495). For more details on migrating to a PEP 495-compliant implementation, see https://pytz-deprecation-shim.readthedocs.io/en/latest/migration.html
  date_obj = stz.localize(date_obj)
```
Ok, there's certainly something going on between frontend and backend values here.  Maybe @shamoon can help me understand how the frontend handles dates.

When I put the document in first, I get the following values:

Creation date from parse_date: `2012-12-12 00:00:00+01:00`  (this is good, the date was located in the text)
Database: `2012-12-11 23:00:00 UTC`  (also makes sense, UTC is 2 hours behind Europe/Berlin
Filename: `20121212_2012-12-12...` (also good, this isn't UTC)
GET: `2012-12-12T00:00:00+01:00` (Seems good, converted UTC back to Europe/Berlin)
Displayed: `12/11/2012` (not so good, went back to UTC maybe?)

I then edit the date back to `12/12/2012`.

The sent payload has `created: "2012-12-12T23:00:00.000Z"`  (sent as UTC instead of local time?)

Database: `2012-12-12 23:00:00 UTC` (matches what the frontend sent)
GET: `created: "2012-12-13T00:00:00+01:00"` (database UTC converted to Europe/Berlin)
File name: `20121213_2012-12-13....`

Is the frontend also doing some amount of date conversions?  Which ends up de-timezoning the date?
I have been following along with these issues waiting to see if the frontend is involved in any way...

The front end does localize dates see https://github.com/paperless-ngx/paperless-ngx/blob/b034e972b031dff50c81964c8b98d14c899dee29/src-ui/src/app/pipes/custom-date.pipe.ts#L26 , the only recent changes to this was https://github.com/paperless-ngx/paperless-ngx/pull/654 which was designed to _retain_ TZ information which was being lost because the datepicker UI component has no awareness of time.

I will play with the example above when I have some time (tonight or tomorrow) and see if I can pin it down further. @stumpylog in the above is your frontend timezone set to Europe/Berlin to match the backend?
Yes, this is with both TZ: "Europe/Berlin" and PAPERLESS_TIME_ZONE: "Europe/Berlin" in my environment
@stumpylog I think your frontend is using US dates, I think that because your frontend is showing `mm/dd/yyyy`, which is US format instead of German `dd.mm.yyyy`. If you change the frontend to use German (or system language as long as its set as such) it displays `11.12.2012`. In either case if I leave the date alone its unchanged in the DB (which suggests #654 works btw).

I think thats correct, no?
I've changed the date display to German/medium, which is hopefully enough.  I still see `"2012-12-12T00:00:00+01:00"` coming back from the GET, then `11.12.2012`, which looks incorrect to me still.  Shouldn't it be `12.12.2012` still?

I do think I've also located something in the backend though, I'm looking into it further as well.
Oh you’re right (of course). Ok lemme look again. Also next time let’s choose a less confusing example haha

Dates!
I know, dates are the most annoying thing.  Then add in timezones, it's even worse!

I've located the issue on the backend side anyway.  It looks like when a rename or export happens, the database time is used, so UTC timezone.  The fix would instead be localizing the date.
Yeah totally. That’s great. 

FYI, I think the frontend issue here is only happening if the locale is not the same as your system. It’s not such a common situation which is why I don’t think most people are seeing it and I don’t think it’s relevant to this bug. Obviously does need a fix, I’ll take care of that of course

Resolving patch:
diff --git a/src/documents/file_handling.py b/src/documents/file_handling.py
--- a/src/documents/file_handling.py
+++ b/src/documents/file_handling.py
@@ -6,6 +6,7 @@
 import pathvalidate
 from django.conf import settings
 from django.template.defaultfilters import slugify
+from django.utils import timezone
 
 
 logger = logging.getLogger("paperless.filehandling")
@@ -158,18 +159,22 @@ def generate_filename(doc, counter=0, append_gpg=True, archive_filename=False):
             else:
                 asn = "none"
 
+            # Convert UTC database date to localized date
+            local_added = timezone.localdate(doc.added)
+            local_created = timezone.localdate(doc.created)
+
             path = settings.PAPERLESS_FILENAME_FORMAT.format(
                 title=pathvalidate.sanitize_filename(doc.title, replacement_text="-"),
                 correspondent=correspondent,
                 document_type=document_type,
-                created=datetime.date.isoformat(doc.created),
-                created_year=doc.created.year if doc.created else "none",
-                created_month=f"{doc.created.month:02}" if doc.created else "none",
-                created_day=f"{doc.created.day:02}" if doc.created else "none",
-                added=datetime.date.isoformat(doc.added),
-                added_year=doc.added.year if doc.added else "none",
-                added_month=f"{doc.added.month:02}" if doc.added else "none",
-                added_day=f"{doc.added.day:02}" if doc.added else "none",
+                created=datetime.date.isoformat(local_created),
+                created_year=local_created.year,
+                created_month=f"{local_created.month:02}",
+                created_day=f"{local_created.day:02}",
+                added=datetime.date.isoformat(local_added),
+                added_year=local_added.year,
+                added_month=f"{local_added.month:02}",
+                added_day=f"{local_added.day:02}",
                 asn=asn,
                 tags=tags,
                 tag_list=tag_list,
diff --git a/src/documents/models.py b/src/documents/models.py
--- a/src/documents/models.py
+++ b/src/documents/models.py
@@ -208,7 +208,9 @@ class Meta:
         verbose_name_plural = _("documents")
 
     def __str__(self):
-        created = datetime.date.isoformat(self.created)
+
+        # Convert UTC database time to local time
+        created = datetime.date.isoformat(timezone.localdate(self.created))
 
         if self.correspondent and self.title:
             return f"{created} {self.correspondent} {self.title}"


Test patch:
diff --git a/src/documents/tests/test_document_model.py b/src/documents/tests/test_document_model.py
--- a/src/documents/tests/test_document_model.py
+++ b/src/documents/tests/test_document_model.py
@@ -56,34 +56,62 @@ def test_file_name(self):
         doc = Document(
             mime_type="application/pdf",
             title="test",
-            created=timezone.datetime(2020, 12, 25),
+            created=timezone.datetime(2020, 12, 25, tzinfo=zoneinfo.ZoneInfo("UTC")),
         )
         self.assertEqual(doc.get_public_filename(), "2020-12-25 test.pdf")
 
+    @override_settings(
+        TIME_ZONE="Europe/Berlin",
+    )
     def test_file_name_with_timezone(self):
 
+        # See https://docs.djangoproject.com/en/4.0/ref/utils/#django.utils.timezone.now
+        # The default for created is an aware datetime in UTC
+        # This does that, just manually, with a fixed date
+        local_create_date = timezone.datetime(
+            2020,
+            12,
+            25,
+            tzinfo=zoneinfo.ZoneInfo("Europe/Berlin"),
+        )
+
+        utc_create_date = local_create_date.astimezone(zoneinfo.ZoneInfo("UTC"))
+
         doc = Document(
             mime_type="application/pdf",
             title="test",
-            created=timezone.datetime(
-                2020,
-                12,
-                25,
-                0,
-                0,
-                0,
-                0,
-                zoneinfo.ZoneInfo("Europe/Berlin"),
-            ),
+            created=utc_create_date,
         )
+
+        # Ensure the create date would cause an off by 1 if not properly created above
+        self.assertEqual(utc_create_date.date().day, 24)
         self.assertEqual(doc.get_public_filename(), "2020-12-25 test.pdf")
 
+        local_create_date = timezone.datetime(
+            2020,
+            1,
+            1,
+            tzinfo=zoneinfo.ZoneInfo("Europe/Berlin"),
+        )
+
+        utc_create_date = local_create_date.astimezone(zoneinfo.ZoneInfo("UTC"))
+
+        doc = Document(
+            mime_type="application/pdf",
+            title="test",
+            created=utc_create_date,
+        )
+
+        # Ensure the create date would cause an off by 1 in the year if not properly created above
+        self.assertEqual(utc_create_date.date().year, 2019)
+        self.assertEqual(doc.get_public_filename(), "2020-01-01 test.pdf")
+
     def test_file_name_jpg(self):
 
         doc = Document(
             mime_type="image/jpeg",
             title="test",
-            created=timezone.datetime(2020, 12, 25),
+            created=timezone.datetime(2020, 12, 25, tzinfo=zoneinfo.ZoneInfo("UTC")),
         )
         self.assertEqual(doc.get_public_filename(), "2020-12-25 test.jpg")
 
@@ -92,7 +120,7 @@ def test_file_name_unknown(self):
         doc = Document(
             mime_type="application/zip",
             title="test",
-            created=timezone.datetime(2020, 12, 25),
+            created=timezone.datetime(2020, 12, 25, tzinfo=zoneinfo.ZoneInfo("UTC")),
         )
         self.assertEqual(doc.get_public_filename(), "2020-12-25 test.zip")
 
@@ -101,6 +129,6 @@ def test_file_name_invalid_type(self):
         doc = Document(
             mime_type="image/jpegasd",
             title="test",
-            created=timezone.datetime(2020, 12, 25),
+            created=timezone.datetime(2020, 12, 25, tzinfo=zoneinfo.ZoneInfo("UTC")),
         )
         self.assertEqual(doc.get_public_filename(), "2020-12-25 test")