Repository: paperless-ngx/paperless-ngx

Issue:
[BUG] PAPERLESS_CONSUMER_RECURSIVE together with PAPERLESS_CONSUMER_IGNORE_PATTERNS not ignoring recursively
### Description

I am running paperless-ngx on a docker container and my consume folder is mapped as a volume with source from `/mnt/nas/scans`. That consumer folder lives inside a Synology NAS where my scanner saves files directly into various folders. This is the content of `/mnt/nas/scans`:
```bash
Tue Feb 14 10:40:17 ricardo@viserion [0] ~
$ ls -1 /mnt/nas/scans/
banks/
documents/
'@eaDir'/
government/
images/
invoices/
kita/
mail/
medical/
misc/
```

As can be seen above, there exists a folder called `@eaDir`, which is something created by DSM (Synology's NAS OS) to store metadata and some control files, mostly related to indexed content.
Among various other settings, my `docker-compose.env` contains:

- `PAPERLESS_CONSUMER_RECURSIVE=true`
- `PAPERLESS_CONSUMER_IGNORE_PATTERNS=[".DS_STORE/*", "._*", ".stfolder/*", ".stversions/*", ".localized/*", "desktop.ini", "@eaDir/*"]`

The problem is that adding `"@eaDir/*"` to the ignore patterns array doesn't work, since inside `@eaDir` there are multiple subfolders, which will then have the files inside be picked up (see attached file [eaDir_content.txt](https://github.com/paperless-ngx/paperless-ngx/files/10731055/eaDir_content.txt)). Given some of the files are actually thumbnails (so, images), they end up being processed by paperless-ngx and show up in my documents, which is undesired behavior since they not only are very low resolution, but also just a thumbnail of another file that was already correctly processed.

The only way I could have paperless-ngx bypass this limitation was by adding all the various extensions and filename patterns I could find inside the `@eaDir` hierarchy, like so:

`PAPERLESS_CONSUMER_IGNORE_PATTERNS=[".DS_STORE/*", "._*", ".stfolder/*", ".stversions/*", ".localized/*", "desktop.ini", "@eaDir/*", "*.log", "*@SynoEAStream", "*@SynoResource", "*.delete", "*.tmp", "SYNOFILE*", "*THUMB*", "*.cfs", "*.del", "*.fnm", "*.fdt", "*.fdx", "segments*", "*.tis", "*.tii", "*.frq", "*.prx", "*.nrm", "*.tvx", "*.tvd", "*.tvf", "*.lock", "*.gen", ".DS_Store", "@drive*"]`

My problem with this is that it not only is tedious, but especially error prone and doesn't really scale.

Ideally the pattern matching logic should completely ignore everything from within the folder hierarchy after `@eaDir`, which is what I intended by adding `"@eaDir/*"` to the ignore patterns.

Is this behavior by design or something that could be improved on?

Relevant files:
- [docker-compose.env.gz](https://github.com/paperless-ngx/paperless-ngx/files/10731175/docker-compose.env.gz)
- [docker-compose.yml.gz](https://github.com/paperless-ngx/paperless-ngx/files/10731186/docker-compose.yml.gz)


### Steps to reproduce

1. Use a consume folder that has folders with multiple subfolders
2. Configure `PAPERLESS_CONSUMER_RECURSIVE=true`
3. Add one top level folder to PAPERLESS_CONSUMER_IGNORE_PATTERNS (e.g.: `"folder_to_exclude/*"`)
4. Check docker logs and see files inside subfolders of `folder_to_exclude` being picked up for processing

### Webserver logs

```bash
paperless-webserver-1  | [2023-02-14 10:57:56,378] [WARNING] [paperless.management.consumer] Not consuming file /usr/src/paperless/consume/@eaDir/SYNO@.fileindexdb/_1jk.fnm: Unknown file extension.                                                                                
paperless-webserver-1  | [2023-02-14 10:57:56,379] [WARNING] [paperless.management.consumer] Not consuming file /usr/src/paperless/consume/@eaDir/SYNO@.fileindexdb/_1jk.fdt: Unknown file extension.                                                                                
paperless-webserver-1  | [2023-02-14 10:57:56,380] [WARNING] [paperless.management.consumer] Not consuming file /usr/src/paperless/consume/@eaDir/SYNO@.fileindexdb/_1jk.fdx: Unknown file extension.                                                                                
paperless-webserver-1  | [2023-02-14 10:57:56,381] [WARNING] [paperless.management.consumer] Not consuming file /usr/src/paperless/consume/@eaDir/SYNO@.fileindexdb/_1jk.tis: Unknown file extension.                                                                                
paperless-webserver-1  | [2023-02-14 10:57:56,382] [WARNING] [paperless.management.consumer] Not consuming file /usr/src/paperless/consume/@eaDir/SYNO@.fileindexdb/_1jk.tii: Unknown file extension.                                                                                
paperless-webserver-1  | [2023-02-14 10:57:56,383] [WARNING] [paperless.management.consumer] Not consuming file /usr/src/paperless/consume/@eaDir/SYNO@.fileindexdb/_1jk.frq: Unknown file extension.                                                                                
paperless-webserver-1  | [2023-02-14 10:57:56,383] [WARNING] [paperless.management.consumer] Not consuming file /usr/src/paperless/consume/@eaDir/SYNO@.fileindexdb/_1jk.prx: Unknown file extension.                                                                                
paperless-webserver-1  | [2023-02-14 10:57:56,385] [WARNING] [paperless.management.consumer] Not consuming file /usr/src/paperless/consume/@eaDir/SYNO@.fileindexdb/_1jk.nrm: Unknown file extension.                                                                                
paperless-webserver-1  | [2023-02-14 10:57:56,386] [WARNING] [paperless.management.consumer] Not consuming file /usr/src/paperless/consume/@eaDir/SYNO@.fileindexdb/_1jk.tvx: Unknown file extension.                                                                                
paperless-webserver-1  | [2023-02-14 10:57:56,387] [WARNING] [paperless.management.consumer] Not consuming file /usr/src/paperless/consume/@eaDir/SYNO@.fileindexdb/_1jk.tvd: Unknown file extension.                                                                                
(...)
paperless-webserver-1  | [2023-02-14 10:57:56,517] [INFO] [paperless.management.consumer] Adding /usr/src/paperless/consume/documents/@eaDir/document_0005.jpg/SYNOFILE_THUMB_M.jpg to the task queue.                                                                               
paperless-webserver-1  | [2023-02-14 10:57:56,644] [INFO] [paperless.management.consumer] Adding /usr/src/paperless/consume/documents/@eaDir/document_0004.jpg/SYNOFILE_THUMB_S.jpg to the task queue.                                                                               
paperless-webserver-1  | [2023-02-14 10:57:56,658] [INFO] [paperless.management.consumer] Adding /usr/src/paperless/consume/documents/@eaDir/document_0004.jpg/SYNOFILE_THUMB_M.jpg to the task queue.                                                                               
paperless-webserver-1  | [2023-02-14 10:57:56,693] [INFO] [paperless.management.consumer] Adding /usr/src/paperless/consume/images/@eaDir/img_.jpg/SYNOFILE_THUMB_M.jpg to the task queue.                                                                                           
paperless-webserver-1  | [2023-02-14 10:57:56,716] [INFO] [paperless.management.consumer] Adding /usr/src/paperless/consume/images/@eaDir/img_.jpg/SYNOFILE_THUMB_S.jpg to the task queue.                                                                                           
paperless-webserver-1  | [2023-02-14 10:57:56,741] [INFO] [paperless.management.consumer] Adding /usr/src/paperless/consume/images/@eaDir/img_0003.jpg/SYNOFILE_THUMB_M.jpg to the task queue.                                                                                       
paperless-webserver-1  | [2023-02-14 10:57:56,759] [INFO] [paperless.management.consumer] Adding /usr/src/paperless/consume/images/@eaDir/img_0001.jpg/SYNOFILE_THUMB_S.jpg to the task queue.                                                                                       
paperless-webserver-1  | [2023-02-14 10:57:56,782] [INFO] [paperless.management.consumer] Adding /usr/src/paperless/consume/images/@eaDir/img_0009.jpg/SYNOFILE_THUMB_S.jpg to the task queue.                                                                                       
paperless-webserver-1  | [2023-02-14 10:57:56,801] [INFO] [paperless.management.consumer] Adding /usr/src/paperless/consume/images/@eaDir/img_0010.jpg/SYNOFILE_THUMB_S.jpg to the task queue.                                                                                       
paperless-webserver-1  | [2023-02-14 10:57:56,819] [INFO] [paperless.management.consumer] Adding /usr/src/paperless/consume/images/@eaDir/img_0017.jpg/SYNOFILE_THUMB_S.jpg to the task queue.
```


### Browser logs

_No response_

### Paperless-ngx version

1.12.2

### Host OS

Debian GNU/Linux 11 (bullseye) x86_64

### Installation method

Docker - official image

### Browser

_No response_

### Configuration changes

See file attached to description.

### Other

_No response_


Discussion:
Certainly not ideal.  I'll have a look.
Pathlib's match isn't behaving how I expect in this case.  Despite claiming `match` is glob style matching, it's not really.
```python
>>> Path('@eaDir/tmp/test').match("@eaDir/**")
False
>>> Path('@eaDir/tmp').match("@eaDir/*")
True
>>> Path('@eaDir/tmp/').match("@eaDir/*")
True
>>> Path('@eaDir/tmp/test').match("@eaDir/*")
False
```

There's an open cpython issue about why these don't work.  Something to work around.

Resolving patch:
diff --git a/src/documents/management/commands/document_consumer.py b/src/documents/management/commands/document_consumer.py
--- a/src/documents/management/commands/document_consumer.py
+++ b/src/documents/management/commands/document_consumer.py
@@ -1,5 +1,6 @@
 import logging
 import os
+from fnmatch import filter
 from pathlib import Path
 from pathlib import PurePath
 from threading import Event
@@ -7,6 +8,7 @@
 from time import monotonic
 from time import sleep
 from typing import Final
+from typing import Set
 
 from django.conf import settings
 from django.core.management.base import BaseCommand
@@ -25,15 +27,15 @@
 logger = logging.getLogger("paperless.management.consumer")
 
 
-def _tags_from_path(filepath):
-    """Walk up the directory tree from filepath to CONSUMPTION_DIR
+def _tags_from_path(filepath) -> Set[Tag]:
+    """
+    Walk up the directory tree from filepath to CONSUMPTION_DIR
     and get or create Tag IDs for every directory.
+
+    Returns set of Tag models
     """
-    normalized_consumption_dir = os.path.abspath(
-        os.path.normpath(settings.CONSUMPTION_DIR),
-    )
     tag_ids = set()
-    path_parts = Path(filepath).relative_to(normalized_consumption_dir).parent.parts
+    path_parts = Path(filepath).relative_to(settings.CONSUMPTION_DIR).parent.parts
     for part in path_parts:
         tag_ids.add(
             Tag.objects.get_or_create(name__iexact=part, defaults={"name": part})[0].pk,
@@ -43,14 +45,41 @@ def _tags_from_path(filepath):
 
 
 def _is_ignored(filepath: str) -> bool:
-    normalized_consumption_dir = os.path.abspath(
-        os.path.normpath(settings.CONSUMPTION_DIR),
+    """
+    Checks if the given file should be ignored, based on configured
+    patterns.
+
+    Returns True if the file is ignored, False otherwise
+    """
+    filepath = os.path.abspath(
+        os.path.normpath(filepath),
     )
-    filepath_relative = PurePath(filepath).relative_to(normalized_consumption_dir)
-    return any(filepath_relative.match(p) for p in settings.CONSUMER_IGNORE_PATTERNS)
 
+    # Trim out the consume directory, leaving only filename and it's
+    # path relative to the consume directory
+    filepath_relative = PurePath(filepath).relative_to(settings.CONSUMPTION_DIR)
 
-def _consume(filepath):
+    # March through the components of the path, including directories and the filename
+    # looking for anything matching
+    # foo/bar/baz/file.pdf -> (foo, bar, baz, file.pdf)
+    parts = []
+    for part in filepath_relative.parts:
+        # If the part is not the name (ie, it's a dir)
+        # Need to append the trailing slash or fnmatch doesn't match
+        # fnmatch("dir", "dir/*") == False
+        # fnmatch("dir/", "dir/*") == True
+        if part != filepath_relative.name:
+            part = part + "/"
+        parts.append(part)
+
+    for pattern in settings.CONSUMER_IGNORE_PATTERNS:
+        if len(filter(parts, pattern)):
+            return True
+
+    return False
+
+
+def _consume(filepath: str) -> None:
     if os.path.isdir(filepath) or _is_ignored(filepath):
         return
 
@@ -103,7 +132,13 @@ def _consume(filepath):
         logger.exception("Error while consuming document")
 
 
-def _consume_wait_unmodified(file):
+def _consume_wait_unmodified(file: str) -> None:
+    """
+    Waits for the given file to appear unmodified based on file size
+    and modification time.  Will wait a configured number of seconds
+    and retry a configured number of times before either consuming or
+    giving up
+    """
     if _is_ignored(file):
         return
 
diff --git a/src/paperless/settings.py b/src/paperless/settings.py
--- a/src/paperless/settings.py
+++ b/src/paperless/settings.py
@@ -673,7 +673,7 @@ def default_threads_per_worker(task_workers) -> int:
     json.loads(
         os.getenv(
             "PAPERLESS_CONSUMER_IGNORE_PATTERNS",
-            '[".DS_STORE/*", "._*", ".stfolder/*", ".stversions/*", ".localized/*", "desktop.ini"]',  # noqa: E501
+            '[".DS_STORE/*", "._*", ".stfolder/*", ".stversions/*", ".localized/*", "desktop.ini", "@eaDir/*"]',  # noqa: E501
         ),
     ),
 )


Test patch:
diff --git a/src/documents/tests/test_management_consumer.py b/src/documents/tests/test_management_consumer.py
--- a/src/documents/tests/test_management_consumer.py
+++ b/src/documents/tests/test_management_consumer.py
@@ -247,22 +247,85 @@ def test_mac_write(self):
 
     def test_is_ignored(self):
         test_paths = [
-            (os.path.join(self.dirs.consumption_dir, "foo.pdf"), False),
-            (os.path.join(self.dirs.consumption_dir, "foo", "bar.pdf"), False),
-            (os.path.join(self.dirs.consumption_dir, ".DS_STORE", "foo.pdf"), True),
-            (
-                os.path.join(self.dirs.consumption_dir, "foo", ".DS_STORE", "bar.pdf"),
-                True,
-            ),
-            (os.path.join(self.dirs.consumption_dir, ".stfolder", "foo.pdf"), True),
-            (os.path.join(self.dirs.consumption_dir, "._foo.pdf"), True),
-            (os.path.join(self.dirs.consumption_dir, "._foo", "bar.pdf"), False),
+            {
+                "path": os.path.join(self.dirs.consumption_dir, "foo.pdf"),
+                "ignore": False,
+            },
+            {
+                "path": os.path.join(self.dirs.consumption_dir, "foo", "bar.pdf"),
+                "ignore": False,
+            },
+            {
+                "path": os.path.join(self.dirs.consumption_dir, ".DS_STORE", "foo.pdf"),
+                "ignore": True,
+            },
+            {
+                "path": os.path.join(
+                    self.dirs.consumption_dir,
+                    "foo",
+                    ".DS_STORE",
+                    "bar.pdf",
+                ),
+                "ignore": True,
+            },
+            {
+                "path": os.path.join(
+                    self.dirs.consumption_dir,
+                    ".DS_STORE",
+                    "foo",
+                    "bar.pdf",
+                ),
+                "ignore": True,
+            },
+            {
+                "path": os.path.join(self.dirs.consumption_dir, ".stfolder", "foo.pdf"),
+                "ignore": True,
+            },
+            {
+                "path": os.path.join(self.dirs.consumption_dir, ".stfolder.pdf"),
+                "ignore": False,
+            },
+            {
+                "path": os.path.join(
+                    self.dirs.consumption_dir,
+                    ".stversions",
+                    "foo.pdf",
+                ),
+                "ignore": True,
+            },
+            {
+                "path": os.path.join(self.dirs.consumption_dir, ".stversions.pdf"),
+                "ignore": False,
+            },
+            {
+                "path": os.path.join(self.dirs.consumption_dir, "._foo.pdf"),
+                "ignore": True,
+            },
+            {
+                "path": os.path.join(self.dirs.consumption_dir, "my_foo.pdf"),
+                "ignore": False,
+            },
+            {
+                "path": os.path.join(self.dirs.consumption_dir, "._foo", "bar.pdf"),
+                "ignore": True,
+            },
+            {
+                "path": os.path.join(
+                    self.dirs.consumption_dir,
+                    "@eaDir",
+                    "SYNO@.fileindexdb",
+                    "_1jk.fnm",
+                ),
+                "ignore": True,
+            },
         ]
-        for file_path, expected_ignored in test_paths:
+        for test_setup in test_paths:
+            filepath = test_setup["path"]
+            expected_ignored_result = test_setup["ignore"]
             self.assertEqual(
-                expected_ignored,
-                document_consumer._is_ignored(file_path),
-                f'_is_ignored("{file_path}") != {expected_ignored}',
+                expected_ignored_result,
+                document_consumer._is_ignored(filepath),
+                f'_is_ignored("{filepath}") != {expected_ignored_result}',
             )
 
     @mock.patch("documents.management.commands.document_consumer.open")