Repository: paperless-ngx/paperless-ngx

Issue:
[BUG] Database/Filenames get out of sync after multiple bulk-updates
### Description

I noticed issues, that certain documents could no longer be opened and in the logs appeared "has gone" errors. The documents were still in the media tree, but with an outdated name. After renaming them to the expected name, everything was ok again.

On digging further, I noticed that this happens when changing several things that influence the filename (like document type and correspondent) in quick succession.

I could reproduce this on a test installation with all documents on local storage as well, so it is not related to network storage for the media.

In the celery.log, I can see that two bulk update jobs are running simultaneously (as I use two workers). If I use only on worker, then the updates run sequentially and the problem does not happen.

I have multiple workflows which are all checked during each document change, causing the rename to run for a considerable time (30s to several minutes) so that it is likely to start the second bulk update while the first one is still running. Also, the presence of workflows seems essential for the problem to occur. With all workflows diabled, also parallel running bulk updates seem to run fine.

### Steps to reproduce

1. Use a file naming scheme that includes things like correspondent and document type, I use `PAPERLESS_FILENAME_FORMAT='{created_year}/{document_type}/{correspondent}/{asn}-{created_year}{created_month}{created_day}-{correspondent}-{title}'`
2. Use at least two workers `PAPERLESS_TASK_WORKERS=2`
3. Use at least one workflow that will be triggered by at least one of the changes. I have a workflow that triggers on "document updated" for a certain document type "Rechnung" and will add some custom fields in that case.
4. Modify a larger number of documents (at least 200 or so) with at least two updates, e.g. change correspondent to something new and document type to "Rechnung", quickly after each other, so that both changes run simultaneously
5. Wait for celery tasks to finish
6. Check document structure by running document_renamer task or sanitizer/export. Many documents will be logged as "has gone". They still exist under a previous name in the filesystem.

### Webserver logs

```bash
In paperless.log, many logs for the workflow handling are visible:

[2024-05-25 21:15:44,378] [INFO] [paperless.matching] Document did not match Workflow: Felder zu "Rechnung" hinzufügen
[2024-05-25 21:15:44,378] [DEBUG] [paperless.matching] ('Document doc type Arztbrief does not match Rechnung',)

Then also errors start to appear:
[2024-05-25 21:16:06,620] [INFO] [paperless.matching] Document matched WorkflowTrigger 7 from Workflow: Felder zu "Rechnung" hinzufügen
[2024-05-25 21:16:06,620] [INFO] [paperless.handlers] Applying WorkflowAction 7 from Workflow: Felder zu "Rechnung" hinzufügen
[2024-05-25 21:16:06,700] [CRITICAL] [paperless.handlers] Document 2022-12-08 Arzt 20221208-somedocument: File /usr/src/paperless/media/documents/originals/2022/Arztbrief/ADAC/-20221208-ADAC-20221208-somedocument.pdf has gone.
[2024-05-25 21:16:06,700] [WARNING] [paperless.handlers] Exception during file handling:

In the celery.log, no errors are shown for the running tasks:

[2024-05-25 21:14:59,439] [INFO] [celery.worker.strategy] Task documents.tasks.bulk_update_documents[d955c8ed-5950-4d2a-8918-3fefbcd4e3b1] received
[2024-05-25 21:14:59,440] [DEBUG] [celery.pool] TaskPool: Apply <function fast_trace_task at 0x778e0262fc40> (args:('documents.tasks.bulk_update_documents', 'd955c8ed-5950-4d2a-8918-3fefbcd4e3b1', {'lang': 'py', 'task': 'documents.tasks.bulk_update_documents', 'id': 'd955c8ed-5950-4d2a-8918-3fefbcd4e3b1', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'group_index': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'd955c8ed-5950-4d2a-8918-3fefbcd4e3b1', 'parent_id': None, 'argsrepr': '()', 'kwargsrepr': "{'document_ids': [2366, 2367, 2358, 2362, 2365, 2196, 2277, 2307, 2312, 2, 13, 1194, 2360, 14, 2008, 2081, 2084, 2364, 2075, 1264, 2359, 2361, 2339, 2385, 11, 1178, 2005, 2079, 2083, 2086, 2088, 2207, 2027, 2036, 2301, 971, 1275, 1257, 2233, 2066, 2363, 2279, 2280, 2382, 9, 1190, 2286, 2333, 2007, 2082, 2085, 970, 1266, 2013, 2349, 1272, 2318, 7, 1189, 2237, 2028, 2056, 2381, 2006, 2080, 2087, 2089, 969, 1270, 2020, 1910, 2278, 2331, 2335, 978, 5, 1192, 2353, 2214, 2330, 2320, 2208, 1924, 1934, 1969, 1976, 1983, 1984, 1997, 2004, 2022, 2379, 2046, 631, 651, 655, 968, 1271, 1258, 1260, 2334, 949,... kwargs:{})
[2024-05-25 21:15:03,827] [INFO] [celery.worker.strategy] Task documents.tasks.bulk_update_documents[1499cb8a-5fa9-4344-adde-266841cefec7] received
[2024-05-25 21:15:03,828] [DEBUG] [celery.pool] TaskPool: Apply <function fast_trace_task at 0x778e0262fc40> (args:('documents.tasks.bulk_update_documents', '1499cb8a-5fa9-4344-adde-266841cefec7', {'lang': 'py', 'task': 'documents.tasks.bulk_update_documents', 'id': '1499cb8a-5fa9-4344-adde-266841cefec7', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'group_index': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '1499cb8a-5fa9-4344-adde-266841cefec7', 'parent_id': None, 'argsrepr': '()', 'kwargsrepr': "{'document_ids': [2366, 2367, 2358, 2362, 2365, 2196, 2277, 2307, 2312, 2, 13, 1194, 2360, 14, 2008, 2081, 2084, 2364, 2075, 1264, 2359, 2361, 2339, 2385, 11, 1178, 2005, 2079, 2083, 2086, 2088, 2207, 2027, 2036, 2301, 971, 1275, 1257, 2233, 2066, 2363, 2279, 2280, 2382, 9, 1190, 2286, 2333, 2007, 2082, 2085, 970, 1266, 2013, 2349, 1272, 2318, 7, 1189, 2237, 2028, 2056, 2381, 2006, 2080, 2087, 2089, 969, 1270, 2020, 1910, 2278, 2331, 2335, 978, 5, 1192, 2353, 2214, 2330, 2320, 2208, 1924, 1934, 1969, 1976, 1983, 1984, 1997, 2004, 2022, 2379, 2046, 631, 651, 655, 968, 1271, 1258, 1260, 2334, 949,... kwargs:{})
[2024-05-25 21:20:03,756] [INFO] [celery.app.trace] Task documents.tasks.bulk_update_documents[1499cb8a-5fa9-4344-adde-266841cefec7] succeeded in 299.9268529689871s: None
[2024-05-25 21:21:12,338] [INFO] [celery.app.trace] Task documents.tasks.bulk_update_documents[d955c8ed-5950-4d2a-8918-3fefbcd4e3b1] succeeded in 372.89696269703563s: None
```


### Browser logs

_No response_

### Paperless-ngx version

2.8.6

### Host OS

Ubuntu 24.04, current docker, ghcr.io docker images

### Installation method

Docker - official image

### System status

_No response_

### Browser

Firefox current

### Configuration changes

_No response_

### Please confirm the following

- [X] I believe this issue is a bug that affects all users of Paperless-ngx, not something specific to my installation.
- [X] I have already searched for relevant existing issues and discussions before opening this report.
- [X] I have updated the title field above with a concise description.


Discussion:
Renames are run asynchronously, but a file lock prevents multiple renames from operating at once.  Certainly can't reproduce this at all.  I would suspect simply being patient is the solution
I am not sure, what you are referring to with "simply being patient". 

When the above described problem occurs, the affected documents are permanently out of sync. Their contents can no longer being displayed through the web interface. Running document_exporter will fail for all affected documents and also document_rename/document_sanity_checker will fail for those documents. This will not heal over time. 

Additionally, I would not expect those problems to occur even while the rename is still running in the background. For sure not, that documents in the web interface will not display during that time. But as said, this will not fix itself over time.

I assume you are not referring to being more patient between two bulk operations and not starting the second one before the first one finishes? While that most likely would resolve the issue, it seems not practical, as I would have to check celery.log to see if it is really finished. Also, as it is working without the active workflow, I assume there is really a bug with the workflow triggering during those updates.

I already messed up my documents several times by now (before finally understanding the cause) and renaming them back even with the help of a script was rather time consuming.

This morning I did another couple of tests on a clean install and was able to reproduce the issue. It really happens every time.

1. Create a fresh instance with [docker-compose.env.txt](https://github.com/paperless-ngx/paperless-ngx/files/15446756/docker-compose.env.txt) and [docker-compose.yml.txt](https://github.com/paperless-ngx/paperless-ngx/files/15446757/docker-compose.yml.txt) (I had to add ".txt", as otherwise I could not upload the files)
2. Consume some hundreds documents
3. Create document TypeA and TypeB
4. Create correspondent A and B
5. Select all documents and then change correspondent to A, apply and then (quickly) document to TypeA, apply. This takes some seconds to rename a files and works without a problem. Can be repeated back and forth between A/B TypeA/TypeB. Make sure, final type is TypeA and correspondent is A.
6. Create custom field "Field" with type Boolean (I assume the type doesn't matter, not sure the custom field is really needed at all for this to happen)
7. Create a workflow for trigger document updated, has document type TypeB. Actions  Assignment, custom field Field
8. Select all documents and then change correspondent to B, apply, and then (quickly) document to TypeB, apply. This will trigger the problem every single time for me.

Important is, that the first bulk update is still running when the second is started. For me, those updates run about 30s with 500 documents and the workflow active.

You can then witness many errors in paperless.log from the workflow and if you try to open documents, many will fail and a "has gone" erorr is in the logs. Nothing will change afterwards anymore, it will stay in this state. If you ran the above tasks, they will all produce lots of errors and/or fail.

I can't figure out what I could be doing wrong here (aside from assuming, I can also do several bulk changes without waiting in between)...
Hello, 

I did some testing, using 500 documents (just a simple generated pdf named `[1-500].pdf` with the contend `[1-500]` in it). It can be reproduced with less documents, but with more documents its easier to get both changes started without the first already beeing finished (altough 500 is perhaps a bit large ;) )

I did make two runs changing correspondend and type from A to B with different workflows triggering.
Findings (more details below):

## workflow which assigns a tag on document update to documents of `typeB`:

* correspondend, document type are set for all documents
* documents are moved to updated location
* metadata gets updated
* tag gets set for some, but not all documents (only 37 of 500 got `tagB` at the end)
* paperless.log shows some infos about locked database
* celery tasks take a long time to succeed

## workflow which assigns a custom field (boolean in this case), on document update to documents of document `typeB`:

* correspondend, document type are set for all documents
* custom field gets assigned to all documents
* documents get moved to updated location
* metadata does only get updated for some documents
* paperless.log shows erros about gone documents, but no messages regarding locked database
* celery tasks `suceeed` rather fast compared to the tagging workflow

So both routes are not correct finished in the end. But:

The tagging workflow just has missing tags. 

The custom field route leaves you with a different amount of unusable documents since they are expected at a different location then they are really located...

So currently I can only discourage users from using workflows in conjunction with more then one parallel worker, since you can not be sure that all changes get applied correctly...


## Some more information/log messages:
### workflow which assigns a tag on document update to documents of `typeB`:
* no corrupted paths
* some log which indicates that it does something, notices the problems with path, and changes accordingly...
```

[2024-05-26 11:50:07,660] [WARNING] [paperless.handlers] Exception during file handling: database is locked
[2024-05-26 11:50:07,661] [INFO] [paperless.handlers] Restoring previous original path
[2024-05-26 11:50:07,662] [INFO] [paperless.handlers] Restoring previous archive path
[2024-05-26 11:50:07,679] [INFO] [paperless.matching] Document did not match Workflow: assign_tag_B
[2024-05-26 11:50:07,679] [DEBUG] [paperless.matching] ('Document doc type typeA does not match typeB',)
```
* in the end everything has correspondend and document type changed and is in the correct directory, but not all of them have the correct tag assigned. (in the last case only 37 of 500 documents got the tag assigned)
* task returns after a long while
* celery log contains:
```
~/ppl-debug/data/log$ tail -f celery.log
    return executor(sql, params, many, context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 83, in _execute
    self.db.validate_no_broken_transaction()
  File "/usr/local/lib/python3.11/site-packages/django/db/backends/base/base.py", line 531, in validate_no_broken_transaction
    raise TransactionManagementError(
django.db.transaction.TransactionManagementError: An error occurred in the current transaction. You can't execute queries until the end of the 'atomic' block.
[2024-05-26 23:08:34,145] [INFO] [celery.worker.request] Discarding revoked task: paperless_mail.tasks.process_mail_accounts[5c432ca4-89da-41d1-91d0-25c6ed8a3651]
[2024-05-26 23:08:34,164] [INFO] [celery.worker.request] Discarding revoked task: paperless_mail.tasks.process_mail_accounts[33d4b42a-3ef2-4d97-bbde-fcededdcf4d1]
[2024-05-26 23:08:47,702] [INFO] [celery.app.trace] Task documents.tasks.bulk_update_documents[2eadba6c-5ef9-4620-9397-df98b974b04f] succeeded in 947.5444359069998s: None

```

## workflow assigning a custom field
* lots of documents end up with corrupted paths
* documents get updated regarding correspondend, document type and custom field, moved to the correct location, but location in metadata stays at e.g. `2024/typeA/corrA/-20240526-corrA-398.pdf` and thus it can not be found by paperless anymore
```
[2024-05-26 23:21:53,547] [INFO] [paperless.handlers] Applying WorkflowAction 2 from Workflow: assign_field
[2024-05-26 23:21:53,600] [CRITICAL] [paperless.handlers] Document 2024-05-26 corrB 398: File /usr/src/paperless/media/documents/originals/2024/typeA/corrA/-20240526-corrA-398.pdf has gone.
[2024-05-26 23:21:53,600] [WARNING] [paperless.handlers] Exception during file handling:
[2024-05-26 23:21:53,608] [CRITICAL] [paperless.handlers] Document 2024-05-26 corrB 398: File /usr/src/paperless/media/documents/originals/2024/typeA/corrA/-20240526-corrA-398.pdf has gone.
[2024-05-26 23:21:53,609] [WARNING] [paperless.handlers] Exception during file handling:
[2024-05-26 23:21:53,627] [INFO] [paperless.matching] Document matched WorkflowTrigger 2 from Workflow: assign_field

```
* task returns quite fast compared to the tag-workflow
```
[2024-05-26 23:22:31,115] [INFO] [celery.app.trace] Task documents.tasks.bulk_update_documents[f4061281-a4e4-463b-b185-e4380f2452e5] succeeded in 52.36065782700007s: None
[2024-05-26 23:22:34,114] [INFO] [celery.app.trace] Task documents.tasks.bulk_update_documents[51e11990-cd65-44e3-a923-af4497590db8] succeeded in 51.73227215099996s: None

```



Resolving patch:
diff --git a/src/documents/signals/handlers.py b/src/documents/signals/handlers.py
--- a/src/documents/signals/handlers.py
+++ b/src/documents/signals/handlers.py
@@ -363,24 +363,22 @@ class CannotMoveFilesException(Exception):
     pass
 
 
-def validate_move(instance, old_path, new_path):
-    if not os.path.isfile(old_path):
-        # Can't do anything if the old file does not exist anymore.
-        logger.fatal(f"Document {instance!s}: File {old_path} has gone.")
-        raise CannotMoveFilesException
-
-    if os.path.isfile(new_path):
-        # Can't do anything if the new file already exists. Skip updating file.
-        logger.warning(
-            f"Document {instance!s}: Cannot rename file "
-            f"since target path {new_path} already exists.",
-        )
-        raise CannotMoveFilesException
-
-
 @receiver(models.signals.m2m_changed, sender=Document.tags.through)
 @receiver(models.signals.post_save, sender=Document)
 def update_filename_and_move_files(sender, instance: Document, **kwargs):
+    def validate_move(instance, old_path, new_path):
+        if not os.path.isfile(old_path):
+            # Can't do anything if the old file does not exist anymore.
+            msg = f"Document {instance!s}: File {old_path} doesn't exist."
+            logger.fatal(msg)
+            raise CannotMoveFilesException(msg)
+
+        if os.path.isfile(new_path):
+            # Can't do anything if the new file already exists. Skip updating file.
+            msg = f"Document {instance!s}: Cannot rename file since target path {new_path} already exists."
+            logger.warning(msg)
+            raise CannotMoveFilesException(msg)
+
     if not instance.filename:
         # Can't update the filename if there is no filename to begin with
         # This happens when the consumer creates a new document.
@@ -532,6 +530,196 @@ def run_workflow(
     document: Document,
     logging_group=None,
 ):
+    def assignment_action():
+        if action.assign_tags.all().count() > 0:
+            document.tags.add(*action.assign_tags.all())
+
+        if action.assign_correspondent is not None:
+            document.correspondent = action.assign_correspondent
+
+        if action.assign_document_type is not None:
+            document.document_type = action.assign_document_type
+
+        if action.assign_storage_path is not None:
+            document.storage_path = action.assign_storage_path
+
+        if action.assign_owner is not None:
+            document.owner = action.assign_owner
+
+        if action.assign_title is not None:
+            try:
+                document.title = parse_doc_title_w_placeholders(
+                    action.assign_title,
+                    (
+                        document.correspondent.name
+                        if document.correspondent is not None
+                        else ""
+                    ),
+                    (
+                        document.document_type.name
+                        if document.document_type is not None
+                        else ""
+                    ),
+                    (document.owner.username if document.owner is not None else ""),
+                    timezone.localtime(document.added),
+                    (
+                        document.original_filename
+                        if document.original_filename is not None
+                        else ""
+                    ),
+                    timezone.localtime(document.created),
+                )
+            except Exception:
+                logger.exception(
+                    f"Error occurred parsing title assignment '{action.assign_title}', falling back to original",
+                    extra={"group": logging_group},
+                )
+
+        if (
+            (
+                action.assign_view_users is not None
+                and action.assign_view_users.count() > 0
+            )
+            or (
+                action.assign_view_groups is not None
+                and action.assign_view_groups.count() > 0
+            )
+            or (
+                action.assign_change_users is not None
+                and action.assign_change_users.count() > 0
+            )
+            or (
+                action.assign_change_groups is not None
+                and action.assign_change_groups.count() > 0
+            )
+        ):
+            permissions = {
+                "view": {
+                    "users": action.assign_view_users.all().values_list(
+                        "id",
+                    )
+                    or [],
+                    "groups": action.assign_view_groups.all().values_list(
+                        "id",
+                    )
+                    or [],
+                },
+                "change": {
+                    "users": action.assign_change_users.all().values_list(
+                        "id",
+                    )
+                    or [],
+                    "groups": action.assign_change_groups.all().values_list(
+                        "id",
+                    )
+                    or [],
+                },
+            }
+            set_permissions_for_object(
+                permissions=permissions,
+                object=document,
+                merge=True,
+            )
+
+        if action.assign_custom_fields is not None:
+            for field in action.assign_custom_fields.all():
+                if (
+                    CustomFieldInstance.objects.filter(
+                        field=field,
+                        document=document,
+                    ).count()
+                    == 0
+                ):
+                    # can be triggered on existing docs, so only add the field if it doesn't already exist
+                    CustomFieldInstance.objects.create(
+                        field=field,
+                        document=document,
+                    )
+
+    def removal_action():
+        if action.remove_all_tags:
+            document.tags.clear()
+        else:
+            for tag in action.remove_tags.filter(
+                pk__in=list(document.tags.values_list("pk", flat=True)),
+            ).all():
+                document.tags.remove(tag.pk)
+
+        if action.remove_all_correspondents or (
+            document.correspondent
+            and (
+                action.remove_correspondents.filter(
+                    pk=document.correspondent.pk,
+                ).exists()
+            )
+        ):
+            document.correspondent = None
+
+        if action.remove_all_document_types or (
+            document.document_type
+            and (
+                action.remove_document_types.filter(
+                    pk=document.document_type.pk,
+                ).exists()
+            )
+        ):
+            document.document_type = None
+
+        if action.remove_all_storage_paths or (
+            document.storage_path
+            and (
+                action.remove_storage_paths.filter(
+                    pk=document.storage_path.pk,
+                ).exists()
+            )
+        ):
+            document.storage_path = None
+
+        if action.remove_all_owners or (
+            document.owner
+            and (action.remove_owners.filter(pk=document.owner.pk).exists())
+        ):
+            document.owner = None
+
+        if action.remove_all_permissions:
+            permissions = {
+                "view": {
+                    "users": [],
+                    "groups": [],
+                },
+                "change": {
+                    "users": [],
+                    "groups": [],
+                },
+            }
+            set_permissions_for_object(
+                permissions=permissions,
+                object=document,
+                merge=False,
+            )
+        elif (
+            (action.remove_view_users.all().count() > 0)
+            or (action.remove_view_groups.all().count() > 0)
+            or (action.remove_change_users.all().count() > 0)
+            or (action.remove_change_groups.all().count() > 0)
+        ):
+            for user in action.remove_view_users.all():
+                remove_perm("view_document", user, document)
+            for user in action.remove_change_users.all():
+                remove_perm("change_document", user, document)
+            for group in action.remove_view_groups.all():
+                remove_perm("view_document", group, document)
+            for group in action.remove_change_groups.all():
+                remove_perm("change_document", group, document)
+
+        if action.remove_all_custom_fields:
+            CustomFieldInstance.objects.filter(document=document).delete()
+        elif action.remove_custom_fields.all().count() > 0:
+            CustomFieldInstance.objects.filter(
+                field__in=action.remove_custom_fields.all(),
+                document=document,
+            ).delete()
+
     for workflow in (
         Workflow.objects.filter(
             enabled=True,
@@ -552,6 +740,10 @@ def run_workflow(
         .prefetch_related("triggers")
         .order_by("order")
     ):
+        # This can be called from bulk_update_documents, which may be running multiple times
+        # Refresh this so the matching data is fresh and instance fields are re-freshed
+        # Otherwise, this instance might be behind and overwrite the work another process did
+        document.refresh_from_db()
         if matching.document_matches_workflow(
             document,
             workflow,
@@ -565,198 +757,10 @@ def run_workflow(
                 )
 
                 if action.type == WorkflowAction.WorkflowActionType.ASSIGNMENT:
-                    if action.assign_tags.all().count() > 0:
-                        document.tags.add(*action.assign_tags.all())
-
-                    if action.assign_correspondent is not None:
-                        document.correspondent = action.assign_correspondent
-
-                    if action.assign_document_type is not None:
-                        document.document_type = action.assign_document_type
-
-                    if action.assign_storage_path is not None:
-                        document.storage_path = action.assign_storage_path
-
-                    if action.assign_owner is not None:
-                        document.owner = action.assign_owner
-
-                    if action.assign_title is not None:
-                        try:
-                            document.title = parse_doc_title_w_placeholders(
-                                action.assign_title,
-                                (
-                                    document.correspondent.name
-                                    if document.correspondent is not None
-                                    else ""
-                                ),
-                                (
-                                    document.document_type.name
-                                    if document.document_type is not None
-                                    else ""
-                                ),
-                                (
-                                    document.owner.username
-                                    if document.owner is not None
-                                    else ""
-                                ),
-                                timezone.localtime(document.added),
-                                (
-                                    document.original_filename
-                                    if document.original_filename is not None
-                                    else ""
-                                ),
-                                timezone.localtime(document.created),
-                            )
-                        except Exception:
-                            logger.exception(
-                                f"Error occurred parsing title assignment '{action.assign_title}', falling back to original",
-                                extra={"group": logging_group},
-                            )
-
-                    if (
-                        (
-                            action.assign_view_users is not None
-                            and action.assign_view_users.count() > 0
-                        )
-                        or (
-                            action.assign_view_groups is not None
-                            and action.assign_view_groups.count() > 0
-                        )
-                        or (
-                            action.assign_change_users is not None
-                            and action.assign_change_users.count() > 0
-                        )
-                        or (
-                            action.assign_change_groups is not None
-                            and action.assign_change_groups.count() > 0
-                        )
-                    ):
-                        permissions = {
-                            "view": {
-                                "users": action.assign_view_users.all().values_list(
-                                    "id",
-                                )
-                                or [],
-                                "groups": action.assign_view_groups.all().values_list(
-                                    "id",
-                                )
-                                or [],
-                            },
-                            "change": {
-                                "users": action.assign_change_users.all().values_list(
-                                    "id",
-                                )
-                                or [],
-                                "groups": action.assign_change_groups.all().values_list(
-                                    "id",
-                                )
-                                or [],
-                            },
-                        }
-                        set_permissions_for_object(
-                            permissions=permissions,
-                            object=document,
-                            merge=True,
-                        )
-
-                    if action.assign_custom_fields is not None:
-                        for field in action.assign_custom_fields.all():
-                            if (
-                                CustomFieldInstance.objects.filter(
-                                    field=field,
-                                    document=document,
-                                ).count()
-                                == 0
-                            ):
-                                # can be triggered on existing docs, so only add the field if it doesn't already exist
-                                CustomFieldInstance.objects.create(
-                                    field=field,
-                                    document=document,
-                                )
+                    assignment_action()
 
                 elif action.type == WorkflowAction.WorkflowActionType.REMOVAL:
-                    if action.remove_all_tags:
-                        document.tags.clear()
-                    else:
-                        for tag in action.remove_tags.filter(
-                            pk__in=list(document.tags.values_list("pk", flat=True)),
-                        ).all():
-                            document.tags.remove(tag.pk)
-
-                    if action.remove_all_correspondents or (
-                        document.correspondent
-                        and (
-                            action.remove_correspondents.filter(
-                                pk=document.correspondent.pk,
-                            ).exists()
-                        )
-                    ):
-                        document.correspondent = None
-
-                    if action.remove_all_document_types or (
-                        document.document_type
-                        and (
-                            action.remove_document_types.filter(
-                                pk=document.document_type.pk,
-                            ).exists()
-                        )
-                    ):
-                        document.document_type = None
-
-                    if action.remove_all_storage_paths or (
-                        document.storage_path
-                        and (
-                            action.remove_storage_paths.filter(
-                                pk=document.storage_path.pk,
-                            ).exists()
-                        )
-                    ):
-                        document.storage_path = None
-
-                    if action.remove_all_owners or (
-                        document.owner
-                        and (action.remove_owners.filter(pk=document.owner.pk).exists())
-                    ):
-                        document.owner = None
-
-                    if action.remove_all_permissions:
-                        permissions = {
-                            "view": {
-                                "users": [],
-                                "groups": [],
-                            },
-                            "change": {
-                                "users": [],
-                                "groups": [],
-                            },
-                        }
-                        set_permissions_for_object(
-                            permissions=permissions,
-                            object=document,
-                            merge=False,
-                        )
-                    elif (
-                        (action.remove_view_users.all().count() > 0)
-                        or (action.remove_view_groups.all().count() > 0)
-                        or (action.remove_change_users.all().count() > 0)
-                        or (action.remove_change_groups.all().count() > 0)
-                    ):
-                        for user in action.remove_view_users.all():
-                            remove_perm("view_document", user, document)
-                        for user in action.remove_change_users.all():
-                            remove_perm("change_document", user, document)
-                        for group in action.remove_view_groups.all():
-                            remove_perm("view_document", group, document)
-                        for group in action.remove_change_groups.all():
-                            remove_perm("change_document", group, document)
-
-                    if action.remove_all_custom_fields:
-                        CustomFieldInstance.objects.filter(document=document).delete()
-                    elif action.remove_custom_fields.all().count() > 0:
-                        CustomFieldInstance.objects.filter(
-                            field__in=action.remove_custom_fields.all(),
-                            document=document,
-                        ).delete()
+                    removal_action()
 
             document.save()
 


Test patch: