Repository: paperless-ngx/paperless-ngx

Issue:
[BUG] duplicate key value violates unique constraint "documents_customfieldinstance_unique_document_field"
### Description

I have 3 custom fields of type "Document Link" in a Consumption template. Since the current version, the following error message appears and the document is not imported.

### Steps to reproduce

1. Creating a custom attribute with the type "Document Link"
2. Creating a template with the custom attribute
3. Importing a new file via the consum directory 
4. Template match the new file
5. Import fails

### Webserver logs

```bash
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 477, in trace_task
    R = retval = fun(*args, **kwargs)
                 ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 760, in __protected_call__
    return self.run(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/paperless/src/documents/tasks.py", line 167, in consume_file
    document = Consumer().try_consume_file(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/paperless/src/documents/consumer.py", line 577, in try_consume_file
    self._fail(
  File "/usr/src/paperless/src/documents/consumer.py", line 115, in _fail
    raise ConsumerError(f"{self.filename}: {log_message or message}") from exception

documents.consumer.ConsumerError: ASN231228001_[ar].pdf: The following error occurred while storing document ASN231228001_[ar].pdf after parsing: duplicate key value violates unique constraint "documents_customfieldinstance_unique_document_field"

DETAIL:  Key (document_id, field_id)=(1, 1) already exists.
```


### Browser logs

```bash
[2023-12-28 15:37:48,629] [INFO] [paperless.management.consumer] Using inotify to watch directory for changes: /usr/src/paperless/consume
[2023-12-28 16:05:00,040] [DEBUG] [paperless.classifier] Document classification model does not exist (yet), not performing automatic matching.
[2023-12-28 16:05:00,043] [WARNING] [paperless.tasks] Classifier error: No training data available.
[2023-12-28 16:16:58,574] [INFO] [paperless.management.consumer] Adding /usr/src/paperless/consume/ASN231228001_[ar].pdf to the task queue.
[2023-12-28 16:16:58,681] [DEBUG] [paperless.barcodes] Scanning for barcodes using ZXING
[2023-12-28 16:17:02,426] [DEBUG] [paperless.barcodes] Upscaling image by 2.0 for better barcode detection
[2023-12-28 16:17:07,894] [DEBUG] [paperless.barcodes] Barcode of type BarcodeFormat.QRCode found: ASN231228001
[2023-12-28 16:17:07,923] [DEBUG] [paperless.barcodes] Upscaling image by 2.0 for better barcode detection
[2023-12-28 16:17:19,043] [DEBUG] [paperless.barcodes] Found ASN Barcode: ASN231228001
[2023-12-28 16:17:19,043] [DEBUG] [paperless.barcodes] Found ASN Barcode: ASN231228001
[2023-12-28 16:17:19,043] [INFO] [paperless.tasks] Found ASN in barcode: 231228001
[2023-12-28 16:17:19,087] [INFO] [paperless.matching] Document matched template Arztrechnung
[2023-12-28 16:17:19,100] [INFO] [paperless.matching] Document matched template Rechnung
[2023-12-28 16:17:19,107] [INFO] [paperless.matching] Document did not match template Zahlungserinnerung
[2023-12-28 16:17:19,107] [DEBUG] [paperless.matching] Document filename ASN231228001_[ar].pdf does not match *[ze]*
[2023-12-28 16:17:19,107] [INFO] [paperless.matching] Document matched template Mahnung
[2023-12-28 16:17:19,150] [INFO] [paperless.consumer] Consuming ASN231228001_[ar].pdf
[2023-12-28 16:17:19,164] [DEBUG] [paperless.consumer] Detected mime type: application/pdf
[2023-12-28 16:17:19,167] [DEBUG] [paperless.consumer] Parser: RasterisedDocumentParser
[2023-12-28 16:17:19,169] [DEBUG] [paperless.consumer] Parsing ASN231228001_[ar].pdf...
[2023-12-28 16:17:19,307] [DEBUG] [paperless.parsing.tesseract] Calling OCRmyPDF with args: {'input_file': PosixPath('/tmp/paperless/paperless-ngx4ifi0ip2/ASN231228001_[ar].pdf'), 'output_file': PosixPath('/tmp/paperless/paperless-6q98pv8s/archive.pdf'), 'use_threads': True, 'jobs': 4, 'language': 'deu', 'output_type': 'pdfa', 'progress_bar': False, 'color_conversion_strategy': 'RGB', 'skip_text': True, 'clean': True, 'deskew': True, 'rotate_pages': True, 'rotate_pages_threshold': 12.0, 'sidecar': PosixPath('/tmp/paperless/paperless-6q98pv8s/sidecar.txt')}
[2023-12-28 16:17:55,539] [DEBUG] [paperless.parsing.tesseract] Using text from sidecar file
[2023-12-28 16:17:55,540] [DEBUG] [paperless.consumer] Generating thumbnail for ASN231228001_[ar].pdf...
[2023-12-28 16:17:55,543] [DEBUG] [paperless.parsing] Execute: convert -density 300 -scale 500x5000> -alpha remove -strip -auto-orient /tmp/paperless/paperless-6q98pv8s/archive.pdf[0] /tmp/paperless/paperless-6q98pv8s/convert.webp
[2023-12-28 16:17:59,548] [DEBUG] [paperless.classifier] Document classification model does not exist (yet), not performing automatic matching.
[2023-12-28 16:17:59,551] [DEBUG] [paperless.consumer] Saving record to database
[2023-12-28 16:17:59,551] [DEBUG] [paperless.consumer] Creation date from parse_date: 2014-10-14 00:00:00+00:00
[2023-12-28 16:17:59,685] [ERROR] [paperless.consumer] The following error occurred while storing document ASN231228001_[ar].pdf after parsing: duplicate key value violates unique constraint "documents_customfieldinstance_unique_document_field"
```


### Paperless-ngx version

2.2.0

### Host OS

Ubuntu 22.04

### Installation method

Docker - official image

### Browser

_No response_

### Configuration changes

_No response_

### Other

I have performed a new installation of Paperless-NGX.

### Please confirm the following

- [X] I believe this issue is a bug that affects all users of Paperless-ngx, not something specific to my installation.
- [X] I have already searched for relevant existing issues and discussions before opening this report.
- [X] I have updated the title field above with a concise description.


Discussion:
Please provide the full logs.  You only provided the last exception which isn't helpful in locating a problem.
Hmm, maybe an issue with the symmetrical links thing
[_paperless_logs.txt](https://github.com/paperless-ngx/paperless-ngx/files/13788058/_paperless_logs.txt)

I think I might see the issue
Actually, not sure I do.

Can you post a screenshot of the relevant template here?

For example this is from a template which adds two doc link fields
<img width="593" alt="Screenshot 2023-12-28 at 8 48 51 AM" src="https://github.com/paperless-ngx/paperless-ngx/assets/4887959/321cbcfc-8329-424c-aba6-55f88de9d96d">

Also, just to check, is this a real install or just a test one? I.e. you have a bunch of documents, yes?
Enclosed is the required information:
![Screenshot_20231228_175919](https://github.com/paperless-ngx/paperless-ngx/assets/59511540/0e2b62a7-598d-471b-b434-dfaff0b090a0)



> Also, just to check, is this a real install or just a test one? I.e. you have a bunch of documents, yes?

At the moment it is still a test installation but I wanted to transfer about 1500 documents from my other archive (ecodms) to this one.
Thanks

Oh, one other thing. There are three templates being applied, do any other ones assign the custom field?
```
2023-12-28T16:17:19.087957877Z [2023-12-28 16:17:19,087] [INFO] [paperless.matching] Document matched template Arztrechnung
2023-12-28T16:17:19.100700279Z [2023-12-28 16:17:19,100] [INFO] [paperless.matching] Document matched template Rechnung
2023-12-28T16:17:19.107705997Z [2023-12-28 16:17:19,107] [INFO] [paperless.matching] Document did not match template Zahlungserinnerung
2023-12-28T16:17:19.108175800Z [2023-12-28 16:17:19,107] [INFO] [paperless.matching] Document matched template Mahnung
```
Got it, I think that's the issue
> Got it, I think that's the issue

just a moment... i have a ideas...

Resolving patch:
diff --git a/src/documents/data_models.py b/src/documents/data_models.py
--- a/src/documents/data_models.py
+++ b/src/documents/data_models.py
@@ -55,31 +55,37 @@ def update(self, other: "DocumentMetadataOverrides") -> "DocumentMetadataOverrid
             self.tag_ids = other.tag_ids
         elif other.tag_ids is not None:
             self.tag_ids.extend(other.tag_ids)
+            self.tag_ids = list(set(self.tag_ids))
 
         if self.view_users is None:
             self.view_users = other.view_users
         elif other.view_users is not None:
             self.view_users.extend(other.view_users)
+            self.view_users = list(set(self.view_users))
 
         if self.view_groups is None:
             self.view_groups = other.view_groups
         elif other.view_groups is not None:
             self.view_groups.extend(other.view_groups)
+            self.view_groups = list(set(self.view_groups))
 
         if self.change_users is None:
             self.change_users = other.change_users
         elif other.change_users is not None:
             self.change_users.extend(other.change_users)
+            self.change_users = list(set(self.change_users))
 
         if self.change_groups is None:
             self.change_groups = other.change_groups
         elif other.change_groups is not None:
             self.change_groups.extend(other.change_groups)
+            self.change_groups = list(set(self.change_groups))
 
         if self.custom_field_ids is None:
             self.custom_field_ids = other.custom_field_ids
         elif other.custom_field_ids is not None:
             self.custom_field_ids.extend(other.custom_field_ids)
+            self.custom_field_ids = list(set(self.custom_field_ids))
 
         return self
 


Test patch:
diff --git a/src/documents/tests/test_consumption_templates.py b/src/documents/tests/test_consumption_templates.py
--- a/src/documents/tests/test_consumption_templates.py
+++ b/src/documents/tests/test_consumption_templates.py
@@ -486,3 +486,54 @@ def test_consumption_template_no_match_source(self, m):
         self.assertIn(expected_str, cm.output[0])
         expected_str = f"Document source {DocumentSource.ApiUpload.name} not in ['{DocumentSource.ConsumeFolder.name}', '{DocumentSource.MailFetch.name}']"
         self.assertIn(expected_str, cm.output[1])
+
+    @mock.patch("documents.consumer.Consumer.try_consume_file")
+    def test_consumption_template_repeat_custom_fields(self, m):
+        """
+        GIVEN:
+            - Existing consumption templates which assign the same custom field
+        WHEN:
+            - File that matches is consumed
+        THEN:
+            - Custom field is added the first time successfully
+        """
+        ct = ConsumptionTemplate.objects.create(
+            name="Template 1",
+            order=0,
+            sources=f"{DocumentSource.ApiUpload},{DocumentSource.ConsumeFolder},{DocumentSource.MailFetch}",
+            filter_filename="*simple*",
+        )
+        ct.assign_custom_fields.add(self.cf1.pk)
+        ct.save()
+
+        ct2 = ConsumptionTemplate.objects.create(
+            name="Template 2",
+            order=1,
+            sources=f"{DocumentSource.ApiUpload},{DocumentSource.ConsumeFolder},{DocumentSource.MailFetch}",
+            filter_filename="*simple*",
+        )
+        ct2.assign_custom_fields.add(self.cf1.pk)
+        ct2.save()
+
+        test_file = self.SAMPLE_DIR / "simple.pdf"
+
+        with mock.patch("documents.tasks.async_to_sync"):
+            with self.assertLogs("paperless.matching", level="INFO") as cm:
+                tasks.consume_file(
+                    ConsumableDocument(
+                        source=DocumentSource.ConsumeFolder,
+                        original_file=test_file,
+                    ),
+                    None,
+                )
+                m.assert_called_once()
+                _, overrides = m.call_args
+                self.assertEqual(
+                    overrides["override_custom_field_ids"],
+                    [self.cf1.pk],
+                )
+
+        expected_str = f"Document matched template {ct}"
+        self.assertIn(expected_str, cm.output[0])
+        expected_str = f"Document matched template {ct2}"
+        self.assertIn(expected_str, cm.output[1])