Repository: paperless-ngx/paperless-ngx

Issue:
[BUG] Non-http/https URLs erroneously considered invalid
### Description

When entering the value for a custom field of type "link", URLs using a protocol other than http or https do not pass validation. Any URL that conforms to RFC 3986 should be considered valid, whether or not Paperless-ngx recognizes the protocol.

Examples:

      ftp://ftp.is.co.za/rfc/rfc1808.txt
      http://www.ietf.org/rfc/rfc2396.txt
      obsidian://open?vault=Vault&file=fields%2Fai%2Fai
      ldap://[2001:db8::7]/c=GB?objectClass?one
      mailto:John.Doe@example.com
      news:comp.infosystems.www.servers.unix
      tel:+1-816-555-1212
      telnet://192.0.2.16:80/
      urn:oasis:names:specification:docbook:dtd:xml:4.1.2

See https://datatracker.ietf.org/doc/html/rfc3986



### Steps to reproduce

1. Log into the Web UI of a Paperless-ngx instance containing a custom field of type "link" and at least one document.
2. Add the link custom field to the document.
3. In the custom field input, enter any URL using a protocol other than http or https.
4. Observe the entry is rejected with the error "Enter a valid URL"

### Webserver logs

```bash
n/a
```


### Browser logs

_No response_

### Paperless-ngx version

2.2.1

### Host OS

macOS

### Installation method

Docker - official image

### Browser

Firefox

### Configuration changes

_No response_

### Other

_No response_

### Please confirm the following

- [X] I believe this issue is a bug that affects all users of Paperless-ngx, not something specific to my installation.
- [X] I have already searched for relevant existing issues and discussions before opening this report.
- [X] I have updated the title field above with a concise description.


Discussion:
We use [Django's URLValidator](https://docs.djangoproject.com/en/5.0/ref/validators/#urlvalidator):

https://github.com/paperless-ngx/paperless-ngx/blob/dev/src/documents/serialisers.py#L502

If theres a bug in their implementation perhaps you should raise an issue at that project. Also, I think some of those URLs work fine, e.g.

<img width="1521" alt="Screenshot 2023-12-29 at 12 52 51 AM" src="https://github.com/paperless-ngx/paperless-ngx/assets/4887959/93766c73-dd21-450c-8b9c-1a1e2b29db15">

@shamoon I appreciate your quick read of this issue -- but after reviewing [Django's URLValidator](https://docs.djangoproject.com/en/5.0/ref/validators/#urlvalidator), it seems that Paperless-ngx has simply chosen to not use the full set of features provided by Django that would permit Paperless-ngx to validate the "full list of valid URI schemes".

```
class URLValidator(schemes=None, regex=None, message=None, code=None)
```
```
schemes:
    URL/URI scheme list to validate against. If not provided, the default list is ['http', 'https', 'ftp', 'ftps']. As a reference, the IANA website provides a full list of [valid URI schemes](https://www.iana.org/assignments/uri-schemes/uri-schemes.xhtml).
```

According to Django's docs, by ignoring the `schemes` parameter, Paperless-ngx is not respecting the "full list of VALID URI schemes".

To address your suggestion -- there doesn't appear to be any bug in Django's implementation, so it wouldn't make sense to engage that team. However, Paperless-ngx is misusing Django's URLValidator by representing to the user that any valid URL will be accepted, then explicitly telling URLValidator to ignore the majority of valid URLs.

If you're opposed to considering the above a bug, then the bug would be the validation message displayed to the user, misidentifying valid URLs as "invalid". In that case, the message should read "Enter a valid http, https, ftp, or ftps URL" to make it clear that other protocols will be rejected.

So, which is the defect -- rejection of valid URLs, or claiming to respect valid URLs? I suppose either one could qualify.

I would just comment that it seems like an odd choice to intentionally use Django URLValidator in a way that deprives your users of useful functionality, fully supported by Django, with the IANA website providing a full list of [valid URI schemes](https://www.iana.org/assignments/uri-schemes/uri-schemes.xhtml).

Opening up these additional protocols does not affect the document management workflow at all, it only empowers your users with added flexibility for free. Shouldn't the user be in charge of which protocols they want their links to use? I guess that's the question.

Ah, I didn't realize the default settings for URLValidaator limit the protocol, thanks for pointing that out. I guess maybe it’s sort of a feature request but let’s not quibble, I’ll be happy to change it
Thank you @shamoon for your openness to examining this further, and for all of your amazing efforts on this project.

I'm not 100% sure how to advise on expanding the set of accepted protocols, though. Maybe you have some thoughts.

A common use case for additional protocols will be linkage of documents in Paperless-ngx to entries in knowledge management systems.

For example, you can see that IANA lists "onenote://" as a valid URI scheme.

But I use Obsidian, which uses the "obsidian://" URI scheme, which is not listed by IANA. So if you simply use IANA's list, many users will benefit, but my specific requirement won't be satisfied.

Likewise, I'm not seeing a way to make URLValidator accept "any" URI scheme.

So maybe the answer is a new optional setting, where the user may specify the comma-separated list of protocols (beyond http/https/ftp/ftps) they wish to use?

Also - would never dream of quibbling :) Just want to make sure you have all the info. But as a noob in an extremely important and respected project, I will concede anything at the drop of a hat :)



Resolving patch:
diff --git a/src/documents/serialisers.py b/src/documents/serialisers.py
--- a/src/documents/serialisers.py
+++ b/src/documents/serialisers.py
@@ -2,6 +2,7 @@
 import math
 import re
 import zoneinfo
+from decimal import Decimal
 
 import magic
 from celery import states
@@ -9,7 +10,9 @@
 from django.contrib.auth.models import Group
 from django.contrib.auth.models import User
 from django.contrib.contenttypes.models import ContentType
-from django.core.validators import URLValidator
+from django.core.validators import DecimalValidator
+from django.core.validators import MaxLengthValidator
+from django.core.validators import integer_validator
 from django.utils.crypto import get_random_string
 from django.utils.text import slugify
 from django.utils.translation import gettext as _
@@ -41,6 +44,7 @@
 from documents.parsers import is_mime_type_supported
 from documents.permissions import get_groups_with_only_permission
 from documents.permissions import set_permissions_for_object
+from documents.validators import uri_validator
 
 
 # https://www.django-rest-framework.org/api-guide/serializers/#example
@@ -489,17 +493,29 @@ def get_value(self, obj: CustomFieldInstance):
 
     def validate(self, data):
         """
-        For some reason, URLField validation is not run against the value
-        automatically.  Force it to run against the value
+        Probably because we're kind of doing it odd, validation from the model
+        doesn't run against the field "value", so we have to re-create it here.
+
+        Don't like it, but it is better than returning an HTTP 500 when the database
+        hates the value
         """
         data = super().validate(data)
         field: CustomField = data["field"]
-        if (
-            field.data_type == CustomField.FieldDataType.URL
-            and data["value"] is not None
-            and len(data["value"]) > 0
-        ):
-            URLValidator()(data["value"])
+        if "value" in data and data["value"] is not None:
+            if (
+                field.data_type == CustomField.FieldDataType.URL
+                and len(data["value"]) > 0
+            ):
+                uri_validator(data["value"])
+            elif field.data_type == CustomField.FieldDataType.INT:
+                integer_validator(data["value"])
+            elif field.data_type == CustomField.FieldDataType.MONETARY:
+                DecimalValidator(max_digits=12, decimal_places=2)(
+                    Decimal(str(data["value"])),
+                )
+            elif field.data_type == CustomField.FieldDataType.STRING:
+                MaxLengthValidator(limit_value=128)(data["value"])
+
         return data
 
     def reflect_doclinks(
diff --git a/src/documents/validators.py b/src/documents/validators.py
new file mode 100644
--- /dev/null
+++ b/src/documents/validators.py
@@ -0,0 +1,29 @@
+from urllib.parse import urlparse
+
+from django.core.exceptions import ValidationError
+from django.utils.translation import gettext_lazy as _
+
+
+def uri_validator(value) -> None:
+    """
+    Raises a ValidationError if the given value does not parse as an
+    URI looking thing, which we're defining as a scheme and either network
+    location or path value
+    """
+    try:
+        parts = urlparse(value)
+        if not parts.scheme:
+            raise ValidationError(
+                _(f"Unable to parse URI {value}, missing scheme"),
+                params={"value": value},
+            )
+        elif not parts.netloc and not parts.path:
+            raise ValidationError(
+                _(f"Unable to parse URI {value}, missing net location or path"),
+                params={"value": value},
+            )
+    except Exception as e:
+        raise ValidationError(
+            _(f"Unable to parse URI {value}"),
+            params={"value": value},
+        ) from e


Test patch:
diff --git a/src/documents/tests/test_api_custom_fields.py b/src/documents/tests/test_api_custom_fields.py
--- a/src/documents/tests/test_api_custom_fields.py
+++ b/src/documents/tests/test_api_custom_fields.py
@@ -333,19 +333,17 @@ def test_custom_field_validation(self):
             },
             format="json",
         )
-        from pprint import pprint
 
-        pprint(resp.json())
         self.assertEqual(resp.status_code, status.HTTP_400_BAD_REQUEST)
         self.assertEqual(CustomFieldInstance.objects.count(), 0)
         self.assertEqual(len(doc.custom_fields.all()), 0)
 
-    def test_custom_field_value_validation(self):
+    def test_custom_field_value_url_validation(self):
         """
         GIVEN:
             - Document & custom field exist
         WHEN:
-            - API request to set a field value
+            - API request to set a field value to something which is or is not a link
         THEN:
             - HTTP 400 is returned
             - No field instance is created or attached to the document
@@ -360,6 +358,56 @@ def test_custom_field_value_validation(self):
             name="Test Custom Field URL",
             data_type=CustomField.FieldDataType.URL,
         )
+
+        for value in ["not a url", "file:"]:
+            with self.subTest(f"Test value {value}"):
+                resp = self.client.patch(
+                    f"/api/documents/{doc.id}/",
+                    data={
+                        "custom_fields": [
+                            {
+                                "field": custom_field_url.id,
+                                "value": value,
+                            },
+                        ],
+                    },
+                    format="json",
+                )
+
+                self.assertEqual(resp.status_code, status.HTTP_400_BAD_REQUEST)
+                self.assertEqual(CustomFieldInstance.objects.count(), 0)
+                self.assertEqual(len(doc.custom_fields.all()), 0)
+        resp = self.client.patch(
+            f"/api/documents/{doc.id}/",
+            data={
+                "custom_fields": [
+                    {
+                        "field": custom_field_url.id,
+                        "value": "tel:+1-816-555-1212",
+                    },
+                ],
+            },
+            format="json",
+        )
+
+        self.assertEqual(resp.status_code, status.HTTP_200_OK)
+
+    def test_custom_field_value_integer_validation(self):
+        """
+        GIVEN:
+            - Document & custom field exist
+        WHEN:
+            - API request to set a field value to something not an integer
+        THEN:
+            - HTTP 400 is returned
+            - No field instance is created or attached to the document
+        """
+        doc = Document.objects.create(
+            title="WOW",
+            content="the content",
+            checksum="123",
+            mime_type="application/pdf",
+        )
         custom_field_int = CustomField.objects.create(
             name="Test Custom Field INT",
             data_type=CustomField.FieldDataType.INT,
@@ -370,8 +418,8 @@ def test_custom_field_value_validation(self):
             data={
                 "custom_fields": [
                     {
-                        "field": custom_field_url.id,
-                        "value": "not a url",
+                        "field": custom_field_int.id,
+                        "value": "not an int",
                     },
                 ],
             },
@@ -382,21 +430,77 @@ def test_custom_field_value_validation(self):
         self.assertEqual(CustomFieldInstance.objects.count(), 0)
         self.assertEqual(len(doc.custom_fields.all()), 0)
 
-        self.assertRaises(
-            Exception,
-            self.client.patch,
+    def test_custom_field_value_monetary_validation(self):
+        """
+        GIVEN:
+            - Document & custom field exist
+        WHEN:
+            - API request to set a field value to something not a valid monetary decimal
+        THEN:
+            - HTTP 400 is returned
+            - No field instance is created or attached to the document
+        """
+        doc = Document.objects.create(
+            title="WOW",
+            content="the content",
+            checksum="123",
+            mime_type="application/pdf",
+        )
+        custom_field_money = CustomField.objects.create(
+            name="Test Custom Field MONETARY",
+            data_type=CustomField.FieldDataType.MONETARY,
+        )
+
+        resp = self.client.patch(
             f"/api/documents/{doc.id}/",
             data={
                 "custom_fields": [
                     {
-                        "field": custom_field_int.id,
-                        "value": "not an int",
+                        "field": custom_field_money.id,
+                        # Too many places past decimal
+                        "value": 12.123,
                     },
                 ],
             },
             format="json",
         )
 
+        self.assertEqual(resp.status_code, status.HTTP_400_BAD_REQUEST)
+        self.assertEqual(CustomFieldInstance.objects.count(), 0)
+        self.assertEqual(len(doc.custom_fields.all()), 0)
+
+    def test_custom_field_value_short_text_validation(self):
+        """
+        GIVEN:
+            - Document & custom field exist
+        WHEN:
+            - API request to set a field value to a too long string
+        THEN:
+            - HTTP 400 is returned
+            - No field instance is created or attached to the document
+        """
+        doc = Document.objects.create(
+            title="WOW",
+            content="the content",
+            checksum="123",
+            mime_type="application/pdf",
+        )
+        custom_field_string = CustomField.objects.create(
+            name="Test Custom Field STRING",
+            data_type=CustomField.FieldDataType.STRING,
+        )
+
+        resp = self.client.patch(
+            f"/api/documents/{doc.id}/",
+            data={
+                "custom_fields": [
+                    {"field": custom_field_string.id, "value": "a" * 129},
+                ],
+            },
+            format="json",
+        )
+
+        self.assertEqual(resp.status_code, status.HTTP_400_BAD_REQUEST)
         self.assertEqual(CustomFieldInstance.objects.count(), 0)
         self.assertEqual(len(doc.custom_fields.all()), 0)