Repository: paperless-ngx/paperless-ngx

Issue:
[BUG] Updating document with custom_fields set to null returns 500 response
### Description

Seems like there are no validations in place for the custom_fields property in the request body when `PUT`ting/updating a document. This is not an issue with the web-client, but when using the API.

There should probably be some validation logic for this field (i.e. not null).

(for reference: https://github.com/astubenbord/paperless-mobile/issues/324)

### Steps to reproduce

1. Perform a PUT request of an existing document with the `custom_fields` property set to `null`.
2. Observe 500 response

### Webserver logs

```bash
[2023-12-19 21:41:05,097] [ERROR] [django.request] Internal Server Error: /api/documents/2/
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/asgiref/sync.py", line 534, in thread_handler
    raise exc_info[1]
  File "/usr/local/lib/python3.11/site-packages/django/core/handlers/exception.py", line 42, in inner
    response = await get_response(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/asgiref/sync.py", line 534, in thread_handler
    raise exc_info[1]
  File "/usr/local/lib/python3.11/site-packages/django/core/handlers/base.py", line 253, in _get_response_async
    response = await wrapped_callback(
               ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/asgiref/sync.py", line 479, in __call__
    ret: _R = await loop.run_in_executor(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/asgiref/current_thread_executor.py", line 40, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/asgiref/sync.py", line 538, in thread_handler
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/views/decorators/csrf.py", line 56, in wrapper_view
    return view_func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/rest_framework/viewsets.py", line 125, in view
    return self.dispatch(request, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/rest_framework/views.py", line 509, in dispatch
    response = self.handle_exception(exc)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/rest_framework/views.py", line 469, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/usr/local/lib/python3.11/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
    raise exc
  File "/usr/local/lib/python3.11/site-packages/rest_framework/views.py", line 506, in dispatch
    response = handler(request, *args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/paperless/src/documents/views.py", line 319, in update
    response = super().update(request, *args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/rest_framework/mixins.py", line 68, in update
    self.perform_update(serializer)
  File "/usr/local/lib/python3.11/site-packages/rest_framework/mixins.py", line 78, in perform_update
    serializer.save()
  File "/usr/local/lib/python3.11/site-packages/drf_writable_nested/mixins.py", line 233, in save
    return super(BaseNestedModelSerializer, self).save(**kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/rest_framework/serializers.py", line 207, in save
    self.instance = self.update(self.instance, validated_data)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/paperless/src/documents/serialisers.py", line 530, in update
    super().update(instance, validated_data)
  File "/usr/src/paperless/src/documents/serialisers.py", line 245, in update
    return super().update(instance, validated_data)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/drf_writable_nested/mixins.py", line 291, in update
    self.delete_reverse_relations_if_need(instance, reverse_relations)
  File "/usr/local/lib/python3.11/site-packages/drf_writable_nested/mixins.py", line 342, in delete_reverse_relations_if_need
    current_ids = self._extract_related_pks(field, related_data)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/drf_writable_nested/mixins.py", line 118, in _extract_related_pks
    for d in filter(None, related_data):
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: 'NoneType' object is not iterable
```


### Browser logs

_No response_

### Paperless-ngx version

2.1.3

### Host OS

Ubuntu 20.04

### Installation method

Docker - official image

### Browser

_No response_

### Configuration changes

_No response_

### Other

_No response_

### Please confirm the following

- [X] I believe this issue is a bug that affects all users of Paperless-ngx, not something specific to my installation.
- [X] I have already searched for relevant existing issues and discussions before opening this report.
- [X] I have updated the title field above with a concise description.


Discussion:

Resolving patch:
diff --git a/src/documents/serialisers.py b/src/documents/serialisers.py
--- a/src/documents/serialisers.py
+++ b/src/documents/serialisers.py
@@ -591,7 +591,11 @@ class DocumentSerializer(
     archived_file_name = SerializerMethodField()
     created_date = serializers.DateField(required=False)
 
-    custom_fields = CustomFieldInstanceSerializer(many=True, allow_null=True)
+    custom_fields = CustomFieldInstanceSerializer(
+        many=True,
+        allow_null=False,
+        required=False,
+    )
 
     owner = serializers.PrimaryKeyRelatedField(
         queryset=User.objects.all(),


Test patch:
diff --git a/src/documents/tests/test_api_custom_fields.py b/src/documents/tests/test_api_custom_fields.py
--- a/src/documents/tests/test_api_custom_fields.py
+++ b/src/documents/tests/test_api_custom_fields.py
@@ -400,6 +400,32 @@ def test_custom_field_value_validation(self):
         self.assertEqual(CustomFieldInstance.objects.count(), 0)
         self.assertEqual(len(doc.custom_fields.all()), 0)
 
+    def test_custom_field_not_null(self):
+        """
+        GIVEN:
+            - Existing document
+        WHEN:
+            - API request with custom_fields set to null
+        THEN:
+            - HTTP 400 is returned
+        """
+        doc = Document.objects.create(
+            title="WOW",
+            content="the content",
+            checksum="123",
+            mime_type="application/pdf",
+        )
+
+        resp = self.client.patch(
+            f"/api/documents/{doc.id}/",
+            data={
+                "custom_fields": None,
+            },
+            format="json",
+        )
+
+        self.assertEqual(resp.status_code, status.HTTP_400_BAD_REQUEST)
+
     def test_bidirectional_doclink_fields(self):
         """
         GIVEN: