Repository: archesproject/arches
Pull request #12110: Avoid hand-escaping percent signs in I18n_JSON #12060
Author: login: aarongundel
Created: 2025-05-22T20:21:18Z | Merged: 2025-05-30T22:29:06Z

<!--- Provide a general summary of the Pull Request in the Title above -->
### Types of changes
<!--- Put an `x` in the boxes that apply  -->
-   [x] Bugfix (non-breaking change which fixes an issue)
-   [ ] New feature (non-breaking change which adds functionality)
-   [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)

### Description of Change
<!--- Include a brief description of this Pull Request and reasoning behind it. -->
Change the i18n json code to unpack the %% as %.  This stops the percent sign explosion.  Though it seems like this should also affect 7.6, based on my testing it doesn't.  This probably has something to do with how publishing and such is done in 8 vs 7.  

### Issues Solved
<!--- If this Pull Request solves any issues, list them here, and mark the ticket "In Review" in the pipeline project -->
Closes #12060

### Checklist
<!--- Put an `x` in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code.  -->
-   I targeted one of these branches:
    - [ ] dev/8.1.x (under development): features, bugfixes not covered below
    - [x] dev/8.0.x (main support): regressions, crashing bugs, security issues, major bugs in new features
    - [ ] dev/7.6.x (extended support): major security issues, data loss issues
-   [ ] I added a changelog in arches/releases
-   [ ] I submitted a PR to arches-docs (if appropriate)
-   [ ] Unit tests pass locally with my changes
-   [x] I added tests that prove my fix is effective or that my feature works
-   [x] My test fails on the target branch

#### Accessibility Checklist
<!-- If your changes impacted the following areas, mark the appropriate columns. -->
[Developer Guide](https://arches.readthedocs.io/en/stable/developing/advanced/accessibility/)

| Topic            | Changed | Retested |
| ---------------- | ------- | -------- |
| Color contrast   |         |          |
| Form fields      |         |          |
| Headings         |         |          |
| Links            |         |          |
| Keyboard         |         |          |
| Responsive Design|         |          |
| HTML validation  |         |          |
| Screen reader    |         |          |


#### Ticket Background
*   Sponsored by: <!--- Who is funding this effort? Getty Conservation Institute|Self Funded -->
*   Found by: @ <!--- This could be the person who files the bug, but not always. -->
*   Tested by: @ <!--- Testing is an important step in development. Who tested this? -->
*   Designed by: @ <!--- Who designed this new feature-->

### Further comments

<!--- If this is a relatively large or complex change, kick off the discussion by explaining why you chose the solution you did and what alternatives you considered, etc... -->


Linked issue #12060: % signs in widget configs are ending up double-encoded
% signs are ending up multiply-encoded in graph json export/import

See examples in test fixtures, some of these predate 8.0.x, so doesn't sound like a recent regression:

https://github.com/archesproject/arches/blob/d854446d2f486d2184bc54cd1d5b68369b4f0f84/tests/fixtures/resource_graphs/branch_excel_test.json#L80

https://github.com/archesproject/arches/blob/d854446d2f486d2184bc54cd1d5b68369b4f0f84/tests/fixtures/resource_graphs/SpatialViews_Test_Model.json#L418

https://github.com/archesproject/arches/blob/d854446d2f486d2184bc54cd1d5b68369b4f0f84/tests/fixtures/resource_graphs/Search%20Sort%20Test%20Model.json#L55



This was first noticed here, so also check `placeholder`?
https://github.com/archesproject/arches/blob/d854446d2f486d2184bc54cd1d5b68369b4f0f84/arches/app/utils/data_management/resource_graphs/importer.py#L178-L184
Linked issue comments: user: login: jacobtylerwalls
created_at: 2025-05-21T18:52:28Z
body: @aarongundel reports severe performance problems with this
author_association: MEMBER
user: login: jacobtylerwalls
created_at: 2025-05-21T18:53:19Z
body: My best guess is that one or both of the places where we call `json.dumps(value).replace("%", "%%")` is overly protective for json inserts/updates
author_association: MEMBER
user: login: aarongundel
created_at: 2025-05-21T18:54:38Z
body: To replicate the issues I was seeing, all you need to do is edit a node description and save a few dozen times.  The percent signs in the serialized graph exponentially increase and eventually start causing problems with serialized graphs being passed around in requests (they get too large) so the server actually starts crashing.
author_association: CONTRIBUTOR

Diff:
diff --git a/arches/app/models/fields/i18n.py b/arches/app/models/fields/i18n.py
index afaa258d9f..f563d98036 100644
--- a/arches/app/models/fields/i18n.py
+++ b/arches/app/models/fields/i18n.py
@@ -321,7 +321,7 @@ def as_sql(self, compiler, connection):
                 ELSE %s
                 END
             """
-            params.append(json.dumps(self.to_localized_object()).replace("%", "%%"))
+            params.append(json.dumps(self.to_localized_object()))
 
         return sql, tuple(params)
 
diff --git a/arches/app/models/migrations/12002_percent_sign_fix.py b/arches/app/models/migrations/12002_percent_sign_fix.py
new file mode 100644
index 0000000000..651ad3f839
--- /dev/null
+++ b/arches/app/models/migrations/12002_percent_sign_fix.py
@@ -0,0 +1,28 @@
+from django.db import migrations
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ("models", "11959_unique_node_source_widget"),
+    ]
+
+    forward = r"""
+        UPDATE cards_x_nodes_x_widgets
+        SET config = jsonb_set(
+            config,
+            '{width}',
+            to_jsonb(
+                regexp_replace(config->>'width', '^(\d+)%{2,}$', '\1%', 'g')
+            )
+        )
+        WHERE config ? 'width'
+        AND config->>'width' ~ '^\d+%{2,}$';
+    """
+
+    operations = [
+        migrations.RunSQL(
+            forward,
+            migrations.RunSQL.noop,  # No reverse operation needed
+        ),
+    ]
diff --git a/arches/app/utils/data_management/resource_graphs/importer.py b/arches/app/utils/data_management/resource_graphs/importer.py
index 32671d4828..d9bca6a97f 100644
--- a/arches/app/utils/data_management/resource_graphs/importer.py
+++ b/arches/app/utils/data_management/resource_graphs/importer.py
@@ -179,8 +179,7 @@ def check_default_configs(default_configs, configs):
                             # Comparing the entire object against the database
                             # may fail because the incoming json may differ
                             # slightly from the database representation, e.g.
-                            # 'width': '100%' -> 'width': '100%%' in db (escaped)
-                            # or 'placeholder': 'Enter text'
+                            # 'placeholder': 'Enter text'
                             # -> 'placeholder': {'en': 'Enter text'} in db
                             card_id=card_x_node_x_widget["card_id"],
                             node_id=card_x_node_x_widget["node_id"],
diff --git a/tests/fixtures/resource_graphs/Search Sort Test Model.json b/tests/fixtures/resource_graphs/Search Sort Test Model.json
index a23bf359da..ae7be013e2 100644
--- a/tests/fixtures/resource_graphs/Search Sort Test Model.json	
+++ b/tests/fixtures/resource_graphs/Search Sort Test Model.json	
@@ -52,7 +52,7 @@
                             "en": "Enter text"
                         },
                         "uneditable": false,
-                        "width": "100%%%%%%%%"
+                        "width": "100%"
                     },
                     "id": "9f32c24c-7aa4-477a-857e-362c9b487b0c",
                     "label": {
@@ -271,4 +271,4 @@
         "os": "Linux",
         "os version": "5.15.167.4-microsoft-standard-WSL2"
     }
-}
\ No newline at end of file
+}
diff --git a/tests/fixtures/resource_graphs/SpatialViews_Test_Model.json b/tests/fixtures/resource_graphs/SpatialViews_Test_Model.json
index b1fa8f83d5..455d1fe68d 100644
--- a/tests/fixtures/resource_graphs/SpatialViews_Test_Model.json
+++ b/tests/fixtures/resource_graphs/SpatialViews_Test_Model.json
@@ -415,7 +415,7 @@
                             "en": "Enter text"
                         },
                         "uneditable": false,
-                        "width": "100%%"
+                        "width": "100%"
                     },
                     "id": "ef14873a-6675-4eb9-80d6-c60ffb4e59e5",
                     "label": {
@@ -1326,4 +1326,4 @@
         "os": "Linux",
         "os version": "6.6.22-linuxkit"
     }
-}
\ No newline at end of file
+}
diff --git a/tests/fixtures/resource_graphs/branch_excel_test.json b/tests/fixtures/resource_graphs/branch_excel_test.json
index 33623ddc4c..92f5c9f63d 100644
--- a/tests/fixtures/resource_graphs/branch_excel_test.json
+++ b/tests/fixtures/resource_graphs/branch_excel_test.json
@@ -77,7 +77,7 @@
                             "en": "Enter text"
                         },
                         "uneditable": false,
-                        "width": "100%%"
+                        "width": "100%"
                     },
                     "id": "376d98fb-04e9-40aa-85a5-ee22ba1f3662",
                     "label": {
@@ -318,4 +318,4 @@
         "os": "Darwin",
         "os version": "23.5.0"
     }
-}
\ No newline at end of file
+}
diff --git a/tests/localization/field_tests.py b/tests/localization/field_tests.py
index 6a689fb90d..0f828b4ff8 100644
--- a/tests/localization/field_tests.py
+++ b/tests/localization/field_tests.py
@@ -479,3 +479,14 @@ def test_domain_datatype(self):
         # Apostrophe in "it's" is doubly-escaped so it doesn't close the string
         expected = "jsonb_set('{}'::jsonb, array['en'], '\"it''s a bird\"')"
         self.assertEqual(datatype.i18n_as_sql(domain_value, None, None), expected)
+
+    def test_percent_sign_encoding(self):
+        widget_config = I18n_JSON(
+            {
+                "i18n_properties": ["placeholder"],
+                "placeholder": "Enter text",
+                "width": "100%",
+            }
+        )
+        _, params = widget_config.as_sql(None, None)
+        self.assertEqual(json.loads(params[0])["width"], "100%")


Reviews:
user: login: jacobtylerwalls
body: It seems like this fix is located at loading from the db instead of on saving to the db. It would be nice to prevent the dupes from making it into the db at all.

I'm wondering if the fix is here:
https://github.com/archesproject/arches/blob/ad763385764efb6d19a1c6379ff0683e89c9333d/arches/app/models/fields/i18n.py#L324

since `params` is escaped by the db-adapter, we shouldn't need to hand-escape ourselves. I didn't get as far as to test it because I couldn't repro the issue with your note about editing the node description.

Are you able to confirm if it fixes the issue by removing this replace call?
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-23T17:26:59Z
user: login: jacobtylerwalls
body: Ah, I was editing the node description and checking the node description instead of checking the widget config.

> Are you able to confirm if it fixes the issue by removing this replace call?

Now that I retested, it looks like my suggestion fixes the bad data from ever getting into the db in the first place.

Agree this was harder to reproduce on 7.6, but from #12060 we have examples of this predating v8, so I think we should:
- clean up those fixtures now
- write a migration to fix the data in the database instead of trying to clean it when loading it from the db.
- add a regression test
state: CHANGES_REQUESTED
author_association: MEMBER
submitted_at: 2025-05-27T15:06:54Z
user: login: jacobtylerwalls
state: APPROVED
author_association: MEMBER
submitted_at: 2025-05-27T21:46:12Z
user: login: jacobtylerwalls
state: CHANGES_REQUESTED
author_association: MEMBER
submitted_at: 2025-05-28T17:08:11Z
user: login: aarongundel
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-30T19:53:37Z
user: login: jacobtylerwalls
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-30T22:14:55Z
user: login: jacobtylerwalls
state: APPROVED
author_association: MEMBER
submitted_at: 2025-05-30T22:15:37Z

Review comments:
diff_hunk: @@ -0,0 +1,28 @@
+from django.db import migrations
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ("models", "11848_alter_spatialview_geometrynode"),
path: arches/app/models/migrations/12002_percent_sign_fix.py
user: login: jacobtylerwalls
body: just merged another migration to dev/8, so will need adjusting
created_at: 2025-05-28T16:56:53Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,28 @@
+from django.db import migrations
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ("models", "11848_alter_spatialview_geometrynode"),
+    ]
+
+    forward = """
+        UPDATE cards_x_nodes_x_widgets
path: arches/app/models/migrations/12002_percent_sign_fix.py
user: login: jacobtylerwalls
body: I see the idea is this will eventually trickle down to published graphs.
created_at: 2025-05-28T16:57:07Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,28 @@
+from django.db import migrations
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ("models", "11848_alter_spatialview_geometrynode"),
+    ]
+
+    forward = """
+        UPDATE cards_x_nodes_x_widgets
+        SET config = jsonb_set(
+            config,
+            '{width}',
+            to_jsonb(
+                regexp_replace(config->>'width', '^(\d+)%{2,}$', '\1%', 'g')
path: arches/app/models/migrations/12002_percent_sign_fix.py
user: login: jacobtylerwalls
body: This changed my `"width": "100%"` to `"width": "\u0001%"`. Maybe it's overkill to add this if it's not safe.
created_at: 2025-05-28T17:07:07Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,28 @@
+from django.db import migrations
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ("models", "11848_alter_spatialview_geometrynode"),
+    ]
+
+    forward = """
path: arches/app/models/migrations/12002_percent_sign_fix.py
user: login: jacobtylerwalls
body: ```suggestion
    forward = r"""
```
To avoid:
```py
/Users/jwalls/prj/arches/arches/app/models/migrations/12002_percent_sign_fix.py:10: SyntaxWarning: invalid escape sequence '\d'
  forward = """
```
created_at: 2025-05-28T17:07:47Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,28 @@
+from django.db import migrations
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ("models", "11848_alter_spatialview_geometrynode"),
+    ]
+
+    forward = """
+        UPDATE cards_x_nodes_x_widgets
path: arches/app/models/migrations/12002_percent_sign_fix.py
user: login: aarongundel
body: yep
created_at: 2025-05-30T19:53:36Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,28 @@
+from django.db import migrations
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ("models", "11848_alter_spatialview_geometrynode"),
+    ]
+
+    forward = """
+        UPDATE cards_x_nodes_x_widgets
+        SET config = jsonb_set(
+            config,
+            '{width}',
+            to_jsonb(
+                regexp_replace(config->>'width', '^(\d+)%{2,}$', '\1%', 'g')
path: arches/app/models/migrations/12002_percent_sign_fix.py
user: login: jacobtylerwalls
body: [this change](https://github.com/archesproject/arches/pull/12110#discussion_r2112377359) fixed it for me (on python 3.13)
created_at: 2025-05-30T22:14:55Z
author_association: MEMBER

PR comments:
user: login: aarongundel
created_at: 2025-05-22T20:40:18Z
body: test failures appear unrelated to this PR
author_association: CONTRIBUTOR
user: login: jacobtylerwalls
created_at: 2025-05-23T13:48:17Z
body: > Though it seems like this should also affect 7.6, based on my testing it doesn't.

My hypothesis is that it's the same Django 5.2 bugfix mentioned in 28bb2e5

author_association: MEMBER
user: login: jacobtylerwalls
created_at: 2025-05-23T13:57:45Z
body: I'm finding myself testing this manually. Would you mind adding a test? I can see this having the potential to regress in the future.

Maybe in tests/localization/field_tests.py?
author_association: MEMBER
user: login: aarongundel
created_at: 2025-05-23T20:42:39Z
body: @jacobtylerwalls that is definitely where I found the issue to happen as well.  The change I made undoes that.  One thing to note - the "save save save" bit I mentioned in the description - you actually have to do that _then_ publish.  The saving has a compounding effect, then publishing it causes it to be committed to the serialized graph.  I will check on removing the escape there, though removing it in other places has a detrimental effect on saving to the db as "%"  in postgres denotes a parameter.  That's apparently why this was done in the first place.
author_association: CONTRIBUTOR
user: login: jacobtylerwalls
created_at: 2025-05-27T21:58:35Z
body: > write a migration to fix the data in the database instead of trying to clean it when loading it from the db.

happy to punt on that until we get reports of this in the wild. approved!
author_association: MEMBER
user: login: aarongundel
created_at: 2025-05-30T22:14:53Z
body: rebased on 8 to pull in test fixes.
author_association: CONTRIBUTOR