Repository: paperless-ngx/paperless-ngx

Issue:
[BUG] Trouble with None Matching
### Description

I do not want to use Automatic Matching when creating new documents.

All matching settings are set to "None".

However, new documents still get tags, correspondent and storage path assigned.

### Steps to reproduce

1. create some Tags (here: anyTag, Correspondents and Storage Paths
2. forget to deactivate matching
3. add some documents and assign Tag, Correspondent and Storage Path by hand
4. add further documents
5. change matching to "none"
6. add further documents

### Webserver logs

```bash
[2023-07-22 08:01:52,313] [INFO] [paperless.consumer] Consuming doc00792720150227134755-searchable.pdf
[2023-07-22 08:01:52,316] [DEBUG] [paperless.consumer] Detected mime type: application/pdf
[2023-07-22 08:01:52,318] [DEBUG] [paperless.consumer] Parser: RasterisedDocumentParser
[2023-07-22 08:01:52,321] [DEBUG] [paperless.consumer] Parsing doc00792720150227134755-searchable.pdf...[2023-07-22 08:01:52,423] [DEBUG] [paperless.parsing.tesseract] Calling OCRmyPDF with args: {'input_file': PosixPath('/tmp/paperless/paperless-ngxue8hkfyp/doc00792720150227134755-searchable.pdf'), 'output_file': PosixPath('/tmp/paperless/paperless-3n8lvptu/archive.pdf'), 'use_threads': True, 'jobs': 6, 'language': 'deu', 'output_type': 'pdfa', 'progress_bar': False, 'skip_text': True, 'clean': True, 'deskew': True, 'rotate_pages': True, 'rotate_pages_threshold': 12.0, 'sidecar': PosixPath('/tmp/paperless/paperless-3n8lvptu/sidecar.txt')}[2023-07-22 08:01:53,177] [DEBUG] [paperless.parsing.tesseract] Incomplete sidecar file: discarding.[2023-07-22 08:01:53,211] [DEBUG] [paperless.consumer] Generating thumbnail for doc00792720150227134755-searchable.pdf...[2023-07-22 08:01:53,215] [DEBUG] [paperless.parsing] Execute: convert -density 300 -scale 500x5000> -alpha remove -strip -auto-orient /tmp/paperless/paperless-3n8lvptu/archive.pdf[0] /tmp/paperless/paperless-3n8lvptu/convert.webp
[2023-07-22 08:01:55,364] [DEBUG] [paperless.consumer] Saving record to database
[2023-07-22 08:01:55,364] [DEBUG] [paperless.consumer] Creation date from parse_date: 2015-01-01 00:00:00+01:00
[2023-07-22 08:01:55,544] [INFO] [paperless.handlers] Assigning correspondent anyTag to 2015-01-01 doc00792720150227134755-searchable
[2023-07-22 08:01:55,560] [INFO] [paperless.handlers] Tagging "2015-01-01 anyTag doc00792720150227134755-searchable" with "belege-aktuellerMonat, 2015, 01"[2023-07-22 08:01:55,569] [INFO] [paperless.handlers] Assigning storage path gmbh to 2015-01-01 anyTag doc00792720150227134755-searchable
[2023-07-22 08:01:55,607] [DEBUG] [paperless.filehandling] Document has storage_path 1 ({correspondent}/{tag_list}/{original_name}) set
[2023-07-22 08:01:55,615] [DEBUG] [paperless.filehandling] Document has storage_path 1 ({correspondent}/{tag_list}/{original_name}) set
[2023-07-22 08:01:55,618] [DEBUG] [paperless.consumer] Deleting file /tmp/paperless/paperless-ngxue8hkfyp/doc00792720150227134755-searchable.pdf
[2023-07-22 08:01:55,621] [DEBUG] [paperless.parsing.tesseract] Deleting directory /tmp/paperless/paperless-3n8lvptu
[2023-07-22 08:01:55,622] [INFO] [paperless.consumer] Document 2015-01-01 anyTag doc00792720150227134755-searchable consumption finished
[2023-07-22 08:01:55,887] [INFO] [paperless.consumer] Consuming doc00792720150227134755.pdf
[2023-07-22 08:01:55,889] [DEBUG] [paperless.consumer] Detected mime type: application/pdf
[2023-07-22 08:01:55,890] [DEBUG] [paperless.consumer] Parser: RasterisedDocumentParser
[2023-07-22 08:01:55,893] [DEBUG] [paperless.consumer] Parsing doc00792720150227134755.pdf...[2023-07-22 08:01:55,975] [DEBUG] [paperless.parsing.tesseract] Calling OCRmyPDF with args: {'input_file': PosixPath('/tmp/paperless/paperless-ngxj69zwcls/doc00792720150227134755.pdf'), 'output_file': PosixPath('/tmp/paperless/paperless-hqsfvs6q/archive.pdf'), 'use_threads': True, 'jobs': 6, 'language': 'deu', 'output_type': 'pdfa', 'progress_bar': False, 'skip_text': True, 'clean': True, 'deskew': True, 'rotate_pages': True, 'rotate_pages_threshold': 12.0, 'sidecar': PosixPath('/tmp/paperless/paperless-hqsfvs6q/sidecar.txt')}[2023-07-22 08:02:04,362] [DEBUG] [paperless.parsing.tesseract] Using text from sidecar file
[2023-07-22 08:02:04,362] [DEBUG] [paperless.consumer] Generating thumbnail for doc00792720150227134755.pdf...[2023-07-22 08:02:04,366] [DEBUG] [paperless.parsing] Execute: convert -density 300 -scale 500x5000> -alpha remove -strip -auto-orient /tmp/paperless/paperless-hqsfvs6q/archive.pdf[0] /tmp/paperless/paperless-hqsfvs6q/convert.webp
[2023-07-22 08:02:06,035] [DEBUG] [paperless.consumer] Saving record to database
[2023-07-22 08:02:06,036] [DEBUG] [paperless.consumer] Creation date from parse_date: 2015-01-01 00:00:00+01:00
[2023-07-22 08:02:06,196] [INFO] [paperless.handlers] Assigning correspondent anyTag to 2015-01-01 doc00792720150227134755
[2023-07-22 08:02:06,212] [INFO] [paperless.handlers] Tagging "2015-01-01 anyTag doc00792720150227134755" with "belege-aktuellerMonat, 2015, 01"[2023-07-22 08:02:06,222] [INFO] [paperless.handlers] Assigning storage path gmbh to 2015-01-01 anyTag doc00792720150227134755
[2023-07-22 08:02:06,254] [DEBUG] [paperless.filehandling] Document has storage_path 1 ({correspondent}/{tag_list}/{original_name}) set
[2023-07-22 08:02:06,263] [DEBUG] [paperless.filehandling] Document has storage_path 1 ({correspondent}/{tag_list}/{original_name}) set
[2023-07-22 08:02:06,266] [DEBUG] [paperless.consumer] Deleting file /tmp/paperless/paperless-ngxj69zwcls/doc00792720150227134755.pdf
[2023-07-22 08:02:06,269] [DEBUG] [paperless.parsing.tesseract] Deleting directory /tmp/paperless/paperless-hqsfvs6q
[2023-07-22 08:02:06,270] [INFO] [paperless.consumer] Document 2015-01-01 anyTag doc00792720150227134755 consumption finished
[2023-07-22 08:02:33,916] [DEBUG] [paperless.filehandling] Document has storage_path 1 ({correspondent}/{tag_list}/{original_name}) set
[2023-07-22 08:02:33,927] [DEBUG] [paperless.filehandling] Document has storage_path 1 ({correspondent}/{tag_list}/{original_name}) set
[2023-07-22 08:02:47,102] [DEBUG] [paperless.filehandling] Document has storage_path 1 ({correspondent}/{tag_list}/{original_name}) set
[2023-07-22 08:02:47,111] [DEBUG] [paperless.filehandling] Document has storage_path 1 ({correspondent}/{tag_list}/{original_name}) set
```


### Browser logs

_No response_

### Paperless-ngx version

1.16.5

### Host OS

Archlinux

### Installation method

Docker - official image

### Browser

Brave

### Configuration changes

_No response_

### Other

_No response_


Discussion:
I cant reproduce this so are you certain you correctly set all of the tags / correspondents etc to 'NONE' before consuming the doc?

I dont think this is broken and the code hasn't changed in a very long time...

https://github.com/paperless-ngx/paperless-ngx/blob/7ef81ae10f4079d090599603b787b35e38240677/src/documents/matching.py#L113-L114

https://github.com/paperless-ngx/paperless-ngx/blob/7ef81ae10f4079d090599603b787b35e38240677/src/documents/tests/test_matchables.py#L50
Hi @shamoon,

thank you for double checking the "bug" and the code.

Unfortunately I tried it with a completely new installation (see the handling below) and the automatic matching still occurs.

Hint: I'd assume that the "breakfast" is required to give paperless some time to "calculate" the matching algorithm.

Or: Did i do something wrong?

 1. New installation via "Docker using the Installation Script" with the following information
    - URL:
    - Port: 8001
    - Current time zone [Europe/Berlin]:
    - Database backend (postgres sqlite mariadb) [postgres]:
    - Enable Apache Tika? (yes no) [no]: yes
    - OCR language [eng]: deu
    - User ID [1000]:
    - Group ID [1000]:
    - Target folder [/home/oro/paperless-ngx]: /mnt/m2data/Code/decr/dokr/papeless-ngx
    - Consume folder [/mnt/m2data/Code/decr/dokr/papeless-ngx/consume]: /mnt/m2data/Data/dokr/papeless-ngx/consume
    - Media folder []: /mnt/m2data/Data/dokr/papeless-ngx/media
    - Data folder []: /mnt/m2data/Data/dokr/papeless-ngx/data
    - Database folder []: /mnt/m2data/Data/dokr/papeless-ngx/database/postgres
    - Paperless username [oro]: currentUser
    - Paperless password: anyPassword
    - Paperless password (again): anyPassword
    - Email [currentUser@localhost]:
2. Summary
    Target folder: /mnt/m2data/Code/decr/dokr/papeless-ngx
    Consume folder: /mnt/m2data/Data/dokr/papeless-ngx/consume
    Media folder: /mnt/m2data/Data/dokr/papeless-ngx/media
    Data folder: /mnt/m2data/Data/dokr/papeless-ngx/data
    Database folder: /mnt/m2data/Data/dokr/papeless-ngx/database/postgres

    URL:
    Port: 8001
    Database: postgres
    Tika enabled: yes
    OCR language: deu
    User id: 1000
    Group id: 1000

    Paperless username: currentUser
    Paperless email: currentUser@localhost

    Press any key to install.
3. After installation I changed the export folder in docker-compose.yaml to follow the other folder structure to
    - /mnt/m2data/Data/dokr/papeless-ngx/export:/usr/src/paperless/export
4. New superuser
    > docker-compose run --rm webserver createsuperuser
    - Username (leave blank to use 'paperless'):
    - Email address: paperless@localhost
    - Password: anyPassword123
    - Password (again): anyPassword123
    
    Superuser created successfully.
5. Start paperless-ngx via
    > docker-compose -f docker-compose.yml up
6. New user thinkORo with the following permissions
    - Correspondent: View
    - Document: Add, Change, View
    - DocumentType: View
    - SavedView: Add, Change, View
    - StoragePath: View
    - Note: Add, Change, View
    - Tag: Add, View
7. New Correspondents without any modification (matching as default, no permissions)
    - ltd
    - provate
8. New Tags without any modification (matching as default, no permissions)
    - 01
    - 02
    - 03
    - 2015
    - 2016
    - Accounting
    - WhatEver
9. New Storage Path without any modification (matching as default, no permissions)
    - ltd: {correspondent}/{tag_list}/{original_name}
    - private: {correspondent}/{tag_list}/{original_name}
10. Switch to other user thinkORo to recognize that I don't have a permissions
11. Switch back to admin to set respective permissions to user thinkORo for
    - Correspondents
    - Tags
    - Storage Paths
12. Switch to other user thinkORo to insert new documents in the following way
    - add 24 pdf files
    - add tags, correspondent, storage path to 24 documents
    - add 6 more pdf files -> no automatic matching for Tags, ...
    - add tags, correspondent, storage path to 6 documents
    - add 44 more pdf files
    - Coffee while waiting :-)
    - no automatic matching for Tags, ... for the new 44 pdf files
    - first breakfast :-)
    - add 10 more pdf files -> attention: automatic matching
13. Switch back to admin to remove matching pattern (set to: None - Disable matching)
    - Correspondents
    - Tags
    - Storage Paths
14. Switch to other user thinkORo to insert further documents
    - add 4 pdf files -> attention: **still automatic matching** for Tags, Correspondent and Storage path
> New Correspondents without any modification (matching as default, no permissions)

The default matching is automatic.  If you've really not changed from the default, that's what you get for any matchable item
But as you can see in Step 13 I set everything to "None - Disable matching".

> 13. Switch back to admin to remove matching pattern (set to: None - Disable matching)

![image](https://github.com/paperless-ngx/paperless-ngx/assets/10477663/ce0ff77f-1b17-46ec-b5fa-bb20b791a745)
![image](https://github.com/paperless-ngx/paperless-ngx/assets/10477663/5e31e6c7-f0b0-4d29-8fdf-41db7ffab110)
![image](https://github.com/paperless-ngx/paperless-ngx/assets/10477663/168c9678-9de6-495b-af4f-1ff9322efc87)

Did you try it in the same way as I documented it?
If the items were auto matching, then changed to None, you would need to re-train the classification or wait for it to be trained again on schedule.  Owing to the time consuming nature of training the model, just making a change to a matching item doesn't trigger a retrain.
1. Don't know why but the time isn't my friend in this case. I tried it a minute ago: new documents are still classified. There was no training on schedule.
2. I re-trained the classifier by hand by executing: `document_create_classifier`. After executing I waited for about 30 minutes before I added further documents. Unfortunately with no change: new documents are still classified with the wrong Tags, ...

As mentioned I'm using Docker.

- How can I check and/or configure the scheduling?
- How can I check what's happening while re-training by manual execution? (`document_create_classifier -v [0..3]` doesn't show anything)

Hint: In the `Logs` in the WebUI I cannot find any information about scheduling or re-training.
Okay, the only thing that works is to delete the existing classifiers (Tags, Correspondents, Storage Paths).
I've tested this in my testing environment (see above) and now new documents are not classified any longer.

Unfortunately: In my main system the effort is much higher because of the number of tags.
If I'm following, you had a number of items set to match automatically, but now have changed them all to be None type matches?  And have no items at all which are auto matching still?
As you can see in the screenshot above: it's absolutely correct. No matching, everything is None.

As described in the instructions, I simply re-executed the shell script to test it again. And executed the described steps, which produced the error with me before.

For me it looks like this: the classification was trained at the beginning, after a few documents and some coffees. And now, after everything is set to NONE, the system ignores this NONE and continues to use the trained classification.

As mentioned before: the only way to get rid of this classification it to delete the classifiers like Tags or Correspondents.
 
What happens if you remove the classification model file from the data directory?
I stopped the system, created a data/tmp folder and moved the model to this folder.

I would assume it works as you expected: without any model there is no automatic matching.

Thanks for this advice. And many thanks for all your quick replies. It's not the perfect solution (okay, it is more a workaround) but for now it's working for me.
Ok, I think I see where this can happen.  It's a special case where there were matching items, then no matching items, but the model still exists and gets used.

Resolving patch:
diff --git a/src/documents/matching.py b/src/documents/matching.py
--- a/src/documents/matching.py
+++ b/src/documents/matching.py
@@ -35,7 +35,11 @@ def match_correspondents(document, classifier, user=None):
         correspondents = Correspondent.objects.all()
 
     return list(
-        filter(lambda o: matches(o, document) or o.pk == pred_id, correspondents),
+        filter(
+            lambda o: matches(o, document)
+            or (o.pk == pred_id and o.matching_algorithm == MatchingModel.MATCH_AUTO),
+            correspondents,
+        ),
     )
 
 
@@ -55,7 +59,11 @@ def match_document_types(document, classifier, user=None):
         document_types = DocumentType.objects.all()
 
     return list(
-        filter(lambda o: matches(o, document) or o.pk == pred_id, document_types),
+        filter(
+            lambda o: matches(o, document)
+            or (o.pk == pred_id and o.matching_algorithm == MatchingModel.MATCH_AUTO),
+            document_types,
+        ),
     )
 
 
@@ -71,7 +79,14 @@ def match_tags(document, classifier, user=None):
         tags = Tag.objects.all()
 
     return list(
-        filter(lambda o: matches(o, document) or o.pk in predicted_tag_ids, tags),
+        filter(
+            lambda o: matches(o, document)
+            or (
+                o.matching_algorithm == MatchingModel.MATCH_AUTO
+                and o.pk in predicted_tag_ids
+            ),
+            tags,
+        ),
     )
 
 
@@ -92,7 +107,8 @@ def match_storage_paths(document, classifier, user=None):
 
     return list(
         filter(
-            lambda o: matches(o, document) or o.pk == pred_id,
+            lambda o: matches(o, document)
+            or (o.pk == pred_id and o.matching_algorithm == MatchingModel.MATCH_AUTO),
             storage_paths,
         ),
     )
diff --git a/src/documents/tasks.py b/src/documents/tasks.py
--- a/src/documents/tasks.py
+++ b/src/documents/tasks.py
@@ -65,6 +65,12 @@ def train_classifier():
         and not Correspondent.objects.filter(matching_algorithm=Tag.MATCH_AUTO).exists()
         and not StoragePath.objects.filter(matching_algorithm=Tag.MATCH_AUTO).exists()
     ):
+        logger.info("No automatic matching items, not training")
+        # Special case, items were once auto and trained, so remove the model
+        # and prevent its use again
+        if settings.MODEL_FILE.exists():
+            logger.info(f"Removing {settings.MODEL_FILE} so it won't be used")
+            settings.MODEL_FILE.unlink()
         return
 
     classifier = load_classifier()


Test patch:
diff --git a/src/documents/tests/test_consumer.py b/src/documents/tests/test_consumer.py
--- a/src/documents/tests/test_consumer.py
+++ b/src/documents/tests/test_consumer.py
@@ -561,10 +561,16 @@ def get_filename():
 
     @mock.patch("documents.consumer.load_classifier")
     def testClassifyDocument(self, m):
-        correspondent = Correspondent.objects.create(name="test")
-        dtype = DocumentType.objects.create(name="test")
-        t1 = Tag.objects.create(name="t1")
-        t2 = Tag.objects.create(name="t2")
+        correspondent = Correspondent.objects.create(
+            name="test",
+            matching_algorithm=Correspondent.MATCH_AUTO,
+        )
+        dtype = DocumentType.objects.create(
+            name="test",
+            matching_algorithm=DocumentType.MATCH_AUTO,
+        )
+        t1 = Tag.objects.create(name="t1", matching_algorithm=Tag.MATCH_AUTO)
+        t2 = Tag.objects.create(name="t2", matching_algorithm=Tag.MATCH_AUTO)
 
         m.return_value = MagicMock()
         m.return_value.predict_correspondent.return_value = correspondent.pk