Repository: hydroshare/hydroshare

Issue:
Depositing metadata during publication will fail if resource metadata contains invalid characters
**Description of the bug**
`get_crossref_deposit_xml` will fail if there are non Unicode chars in the abstract, title, etc of the resource.
```
Traceback (most recent call last):
  File "/hydroshare/manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.9/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.9/site-packages/django/core/management/__init__.py", line 413, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.9/site-packages/django/core/management/base.py", line 354, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.9/site-packages/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/hydroshare/hs_core/management/commands/update_crossref_deposit_all_published_resources.py", line 16, in handle
    response = deposit_res_metadata_with_crossref(resource)
  File "/hydroshare/hs_core/hydroshare/resource.py", line 1029, in deposit_res_metadata_with_crossref
    files = {'file': (xml_file_name, res.get_crossref_deposit_xml())}
  File "/hydroshare/hs_core/models.py", line 4077, in get_crossref_deposit_xml
    etree.SubElement(dataset_node, 'description').text = self.metadata.description.abstract
  File "src/lxml/etree.pyx", line 1042, in lxml.etree._Element.text.__set__
  File "src/lxml/apihelpers.pxi", line 748, in lxml.etree._setNodeText
  File "src/lxml/apihelpers.pxi", line 736, in lxml.etree._createTextNode
  File "src/lxml/apihelpers.pxi", line 1541, in lxml.etree._utf8
ValueError: All strings must be XML compatible: Unicode or ASCII, no NULL bytes or control characters
```
[Example run on beta](https://new-hs-ci.hydroshare.org/view/02-Deployment/job/Beta%20deployment/job/beta-hsctl-command/489/console)

**Expected behavior**
We should either sanitize the XML at time of deposit, or better sanitize it in the resource.
We should also notify that metadata contains non-accepted chars (if we don't already)

**Additional information**
Command run using rc-2.12 at 0a5b64cfed01de08478548072d8bc3bd4aae71fe



Discussion:
perhaps sanitize with something like
https://stackoverflow.com/questions/1707890/fast-way-to-filter-illegal-xml-unicode-chars-in-python
example failing resource:
https://www.hydroshare.org/resource/1a7dc5d6b9fa4253bca442341eef500d/
related to #5225
Beyond the approach taken in #5307 which seems to be notification at the time publication is requested I think there should be an effort to prevent invalid characters getting in to the metadata in the first place, perhap with a check when metadata is saved that warns a user about characters that will block publication because the metadata can not be published in crossref. We've discussed a lot the special characters problem, so I am hesitant about opening it up again too broadly. We want to respect special characters in peoples names wherever possible, but we now have a definitive case where using some special characters limits a key goal of hydroshare - namely resource publication, and is thus a good reason for keeping them out (or warning users).  In the example resource given https://www.hydroshare.org/resource/1a7dc5d6b9fa4253bca442341eef500d/ it would be good to determine if the issue is characters in the abstract "δ2H and δ18O" or the expression later on "7 h1." that does not display correctly (and I will be interested to see how github renders these strings cut and paste from the resource abstract)
Actually for the [example resource](https://www.hydroshare.org/resource/1a7dc5d6b9fa4253bca442341eef500d/), the invalid character is a null between the "h" and "1" in the following section of the abstract:
> "greater than 7 **h1**. The soil"

I agree that it would be best to notify users at the time of entry when they have invalid characters, but I thought it would be very difficult to communicate to them programmatically via the UI, what characters in particular are causing the issue. Taking the example resource I provided, we could write an automated check that finds invalid chars and then provides a message to the user containing the text surrounding the invalid char. In this case it is particularly confusing because the offender is null so they don't actually see any characters (likely they copy/pasted from Word or a PDF). I thought it might just be easier to handle as part of the Human in the Loop review, as it might be easiest communicated by a human rather than programmatically. But we can take a go at it!

The alternative is just to programmatically sanitize any offending characters and provide a notification that we have done so. It feels heavy handed, but we would have to eventually remove the chars anyway. I'm going to explore this option further.

Also, I got to wondering why this resource (and others like it) wasn't having issues when we serialize its metadata into RDF in the bagit. We use [rdflib](https://rdflib.readthedocs.io/en/stable/intro_to_parsing.html#saving-rdf) to create the bag xml files, and that serializer will write a "" into the XML when there's an invalid char. Conversely, [our new Crossref XML generation ](https://github.com/hydroshare/hydroshare/blob/rc-2.12/hs_core/models.py#L4077) uses [lxml.etree](https://lxml.de/compatibility.html)

Resolving patch:
diff --git a/hs_core/forms.py b/hs_core/forms.py
--- a/hs_core/forms.py
+++ b/hs_core/forms.py
@@ -10,7 +10,7 @@
 from crispy_forms.bootstrap import Field
 
 from .models import Party, Creator, Contributor, validate_hydroshare_user_id, Relation, Identifier, \
-    FundingAgency, Description
+    FundingAgency, Description, validate_abstract
 
 
 class Helper(object):
@@ -733,7 +733,7 @@ class Meta:
 class AbstractValidationForm(forms.Form):
     """Validate Abstract form with abstract field."""
 
-    abstract = forms.CharField(max_length=5000)
+    abstract = forms.CharField(max_length=5000, validators=[validate_abstract])
 
 
 class RightsValidationForm(forms.Form):
diff --git a/hs_core/migrations/0078_alter_description_abstract.py b/hs_core/migrations/0078_alter_description_abstract.py
new file mode 100644
--- /dev/null
+++ b/hs_core/migrations/0078_alter_description_abstract.py
@@ -0,0 +1,19 @@
+# Generated by Django 3.2.20 on 2024-02-07 17:01
+
+from django.db import migrations, models
+import hs_core.models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('hs_core', '0077_baseresource_spam_allowlisted'),
+    ]
+
+    operations = [
+        migrations.AlterField(
+            model_name='description',
+            name='abstract',
+            field=models.TextField(validators=[hs_core.models.validate_abstract]),
+        ),
+    ]
diff --git a/hs_core/models.py b/hs_core/models.py
--- a/hs_core/models.py
+++ b/hs_core/models.py
@@ -1,5 +1,6 @@
 """Declare critical models for Hydroshare hs_core app."""
 import copy
+import difflib
 import json
 import logging
 import os.path
@@ -745,12 +746,25 @@ class Meta:
         ordering = ['order']
 
 
+def validate_abstract(value):
+    """Validate that an abstract is valid."""
+    err_message = 'The abstract is not valid. It contains characters that are not XML compatible.'
+    if value:
+        try:
+            clean = clean_for_xml(value)
+            assert (len(clean) == len(value))
+            match_ratio = difflib.SequenceMatcher(None, clean.splitlines(), value.splitlines()).ratio()
+            assert (match_ratio == 1.0)
+        except AssertionError:
+            raise ValidationError(err_message)
+
+
 @rdf_terms(DC.description, abstract=DCTERMS.abstract)
 class Description(AbstractMetaDataElement):
     """Define Description metadata element model."""
 
     term = 'Description'
-    abstract = models.TextField()
+    abstract = models.TextField(validators=[validate_abstract])
 
     def __unicode__(self):
         """Return abstract field for unicode representation."""
@@ -761,6 +775,18 @@ class Meta:
 
         unique_together = ("content_type", "object_id")
 
+    @classmethod
+    def create(cls, **kwargs):
+        """Define custom update method for Description model."""
+        validate_abstract(kwargs['abstract'])
+        return super(Description, cls).create(**kwargs)
+
+    @classmethod
+    def update(cls, element_id, **kwargs):
+        """Define custom update method for Description model."""
+        validate_abstract(kwargs['abstract'])
+        super(Description, cls).update(element_id, **kwargs)
+
     @classmethod
     def remove(cls, element_id):
         """Create custom remove method for Description model."""


Test patch:
diff --git a/hs_core/tests/api/native/test_core_metadata.py b/hs_core/tests/api/native/test_core_metadata.py
--- a/hs_core/tests/api/native/test_core_metadata.py
+++ b/hs_core/tests/api/native/test_core_metadata.py
@@ -1401,6 +1401,43 @@ def test_description(self):
             ),
         )
 
+    def test_description_xml(self):
+
+        # test that the resource metadata does not contain abstract
+        self.assertEqual(
+            self.res.metadata.description, None, msg="Abstract exists for the resource"
+        )
+
+        # test adding invalid XML raises exception
+        self.assertRaises(
+            ValidationError,
+            lambda: resource.create_metadata_element(
+                self.res.short_id,
+                "description",
+                abstract="analysis throughput rates greater than 7 h1. The soil dept",
+            ),
+        )
+
+        # create a abstract for the resource
+        resource.create_metadata_element(
+            self.res.short_id, "description", abstract="new abstract for the resource"
+        )
+
+        # attempt update, this should fail
+        self.assertRaises(
+            ValidationError,
+            lambda: resource.update_metadata_element(
+                self.res.short_id,
+                "description",
+                self.res.metadata.description.id,
+                abstract="analysis throughput rates greater than 7 h1. The soil dept",
+            )
+        )
+        # the abstract should remain unchanged
+        self.assertEqual(
+            self.res.metadata.description.abstract, "new abstract for the resource"
+        )
+
     def test_format(self):
         # when a resource is created with no content files, there should not be any formats elements
         # associated with it