Repository: paperless-ngx/paperless-ngx

Issue:
[BUG] advanced searching for created date gives bad results
### Description

Searching for created date in the format "created:20231119" shows documents created one day later.
<img width="640" alt="image" src="https://github.com/paperless-ngx/paperless-ngx/assets/52917795/1e586f2a-cdd2-48c5-8112-ee7ee64f3efc">


### Steps to reproduce

1. Go to Documents
2. Switch to Advanced search
3. Put "created:<date>" (without hyphens as per https://whoosh.readthedocs.io/en/latest/dates.html#parsing-date-queries)
4. All the results have created date one day later.

### Webserver logs

```bash
no logs
```


### Browser logs

_No response_

### Paperless-ngx version

2.0.1

### Host OS

Unraid (Linux)

### Installation method

Docker - official image

### Browser

Brave (Chrome)

### Configuration changes

_No response_

### Other

CET timezone if relevant (UTC+1:00).


Discussion:
I'm pretty sure that doesn't use Whoosh, but the database.  Which isn't going to be incorrect.
> I'm pretty sure that doesn't use Whoosh, but the database. Which isn't going to be incorrect.

According to this it uses whoosh: https://docs.paperless-ngx.com/usage/#basic-usage_searching
I also cannot reproduce this.  created and added match to the documents' values.
Even with setting the timezone as Europe/Brussels (UTC +1), added and created all make sense when searched for
Well, you can see the screenshot above. And I've just reproduced in on another computer with another browser. 
What data is needed to understand what's happening? Any logs I can upload here?
My assumption is the problem is your timezone in the browser or the OS, so the documents do have that date but in the browser the date is getting timezone-ified so it looks wrong to you.

What does that search look like against the api (look in the browser console and open the url like /api/documents/... )

But no, I cant reproduce this
<img width="361" alt="Screenshot 2023-12-01 at 4 46 16 PM" src="https://github.com/paperless-ngx/paperless-ngx/assets/4887959/a8956e99-1bc6-4b11-b058-870579167204">

I used date(Linux) and time(Windows) commands on consoles of Unraid, paperless-ngx and redis dockers, and my Windows system to confirm the time and timezone is the same everywhere. It's the same on all my computers, too. The PAPERLESS_TIME_ZONE is also correct.

I also tried setting different languages and time formats in paperless-ngx. The error still appears.
From above:

> What does that search look like against the api (look in the browser console and open the url like /api/documents/... )
Here is the browser console output of search for created:20231129.

<img width="600" alt="image" src="https://github.com/paperless-ngx/paperless-ngx/assets/52917795/51ef691b-c029-472d-a2bf-5280b18f4e59">

The results I get are from 2023-11-30.

And here's my (correct) time and timezone from the browser console itself:
<img width="374" alt="image" src="https://github.com/paperless-ngx/paperless-ngx/assets/52917795/27ce2047-06c8-4a42-aa5a-70c98cc4db8d">

And here's my environment settings in docker:
<img width="691" alt="image" src="https://github.com/paperless-ngx/paperless-ngx/assets/52917795/9456db95-82fa-47ad-b981-42ef8a8aa200">

> Here is the browser console output of search for created:20231129.

I meant what do the results look like
Here it is:
<img width="868" alt="image" src="https://github.com/paperless-ngx/paperless-ngx/assets/52917795/2d7bf3a2-445c-4e8b-9b51-bb1017d9d832">

Have you tried reindexing? https://docs.paperless-ngx.com/administration/#index
I used document_index reindex and then document_index optimize on console, then restarted the docker. Same error.
I dunno, thing is the code for this is basically straight using whoosh:

https://github.com/paperless-ngx/paperless-ngx/blob/20f27fe32ffe144813d6c69e3326bf230a2da52a/src/documents/index.py#L374-L375

As noted:

- Neither of us can reproduce this, nor with a test
- No one else is reporting it
- We didnt intentionally change anything to do with this
- whoosh hasn't been touched in a while https://github.com/mchaput/whoosh

My guess is theres an alternate explanation, but otherwise I dont see where this is actionable for us, unfortunately.

Resolving patch:
diff --git a/src/documents/index.py b/src/documents/index.py
--- a/src/documents/index.py
+++ b/src/documents/index.py
@@ -25,9 +25,11 @@
 from whoosh.qparser import MultifieldParser
 from whoosh.qparser import QueryParser
 from whoosh.qparser.dateparse import DateParserPlugin
+from whoosh.qparser.dateparse import English
 from whoosh.scoring import TF_IDF
 from whoosh.searching import ResultsPage
 from whoosh.searching import Searcher
+from whoosh.util.times import timespan
 from whoosh.writing import AsyncWriter
 
 # from documents.models import CustomMetadata
@@ -356,6 +358,22 @@ def __getitem__(self, item):
         return page
 
 
+class LocalDateParser(English):
+    def reverse_timezone_offset(self, d):
+        return (d.replace(tzinfo=timezone.get_current_timezone())).astimezone(
+            timezone.utc,
+        )
+
+    def date_from(self, *args, **kwargs):
+        d = super().date_from(*args, **kwargs)
+        if isinstance(d, timespan):
+            d.start = self.reverse_timezone_offset(d.start)
+            d.end = self.reverse_timezone_offset(d.end)
+        else:
+            d = self.reverse_timezone_offset(d)
+        return d
+
+
 class DelayedFullTextQuery(DelayedQuery):
     def _get_query(self):
         q_str = self.query_params["query"]
@@ -371,7 +389,12 @@ def _get_query(self):
             ],
             self.searcher.ixreader.schema,
         )
-        qp.add_plugin(DateParserPlugin(basedate=timezone.now()))
+        qp.add_plugin(
+            DateParserPlugin(
+                basedate=timezone.now(),
+                dateparser=LocalDateParser(),
+            ),
+        )
         q = qp.parse(q_str)
 
         corrected = self.searcher.correct_query(q, q_str)


Test patch:
diff --git a/src/documents/tests/test_api.py b/src/documents/tests/test_api.py
--- a/src/documents/tests/test_api.py
+++ b/src/documents/tests/test_api.py
@@ -964,6 +964,62 @@ def test_search_added_in_last_week_with_timezone_ahead(self):
             # Assert subset in results
             self.assertDictEqual(result, {**result, **subset})
 
+    @override_settings(
+        TIME_ZONE="Europe/Sofia",
+    )
+    def test_search_added_specific_date_with_timezone_ahead(self):
+        """
+        GIVEN:
+            - Two documents added right now
+            - One document added on a specific date
+            - The timezone is behind UTC time (+2)
+        WHEN:
+            - Query for documents added on a specific date
+        THEN:
+            - The one document is returned
+        """
+        d1 = Document.objects.create(
+            title="invoice",
+            content="the thing i bought at a shop and paid with bank account",
+            checksum="A",
+            pk=1,
+        )
+        d2 = Document.objects.create(
+            title="bank statement 1",
+            content="things i paid for in august",
+            pk=2,
+            checksum="B",
+        )
+        d3 = Document.objects.create(
+            title="bank statement 3",
+            content="things i paid for in september",
+            pk=3,
+            checksum="C",
+            # specific time zone aware date
+            added=timezone.make_aware(datetime.datetime(2023, 12, 1)),
+        )
+        # refresh doc instance to ensure we operate on date objects that Django uses
+        # Django converts dates to UTC
+        d3.refresh_from_db()
+
+        with index.open_index_writer() as writer:
+            index.update_document(writer, d1)
+            index.update_document(writer, d2)
+            index.update_document(writer, d3)
+
+        response = self.client.get("/api/documents/?query=added:20231201")
+        results = response.data["results"]
+
+        # Expect 1 document returned
+        self.assertEqual(len(results), 1)
+
+        for idx, subset in enumerate(
+            [{"id": 3, "title": "bank statement 3"}],
+        ):
+            result = results[idx]
+            # Assert subset in results
+            self.assertDictEqual(result, {**result, **subset})
+
     def test_search_added_in_last_month(self):
         """
         GIVEN: