Repository: strictdoc-project/strictdoc (language: python)

Issue:
Extend UID regex and use it in SDocNode
This fixes two things:
- The UID regex is now used in SDocNode to make UID allowed character validation the same as for UIDs from DOCUMENT, SECTION, LINK, ANCHOR.
- The UID regex is extended so that it allows also non latin letters and digits. Dots and whitespaces become valid too, but not as 1st character.

There are multiple approaches how one could fix the issues. This one is the least invasive and most backwards compatible.
- It continues to accept UIDs at arbitrary field positions in custom grammars for backwards compatibility.
- It continues to model a UID as python class SDocNodeField with argument parts of type list. So no code has to be changed at all (besides TextX grammar).

This sdoc works now:
```
[DOCUMENT]
TITLE: Link to UID with whitespace, dot, umlauts, non-latin characters

[REQUIREMENT]
UID: Ölrückstoßabdämpfung - 任何 47.11
STATEMENT: Foo

[TEXT]
STATEMENT: >>>
See [LINK: Ölrückstoßabdämpfung - 任何 47.11]
<<<
```

Fixes #1897.


Discussion / hints:
I'll have to look at failed tests and implement new ones, therefore WIP. You can already start to comment the approach anyways.

EDIT: Fixed tests, added a new one. Most importantly, I changed the regex to also allow non latin characters. I suppose this can be important when converting national standards to sdoc.
Thanks for sharing this very interesting insights.

> With this particular change, Arpeggio will have to jump over UID/non-UID branches when parsing every node field. This should not have much performance overhead but logically, this kind of extra branching is still introduced.

The UID branch is less likely than the generic field branch. UID should have been checked 2nd. Doesn't matter much if we remove the branch soon anyway.

> ```python
> FIELD_VALIDATORS = {
>     "UID": validate_uid,
>     ...
> }
> ```
> 
> and then the `validate_requirement_field` would simply call:
> 
> ```python
> field_validation = FIELD_VALIDATORS.get(field_name, None)
> if field_validation is not None:
>     field_validation(node)
> ```

Looks good. We'll need a function similar to `validate_requirement_field` to check UIDs of DocumentConfig and SDocSection the same way.

> Having this all written, I think it is best if we move the UID regex validation to the SDocValidator right away, introducing the `FIELD_VALIDATORS` and field-specific validation functions. The half of your patch will have to be removed, and a unit test catching a validation for invalid UIDs will have to be written.

This is well reasoned. I'm glad you merged the PR anyway as it allows me to do some immediate show cases.
 
> Let me know if you see any disadvantages of not doing the validation in `validate_requirement_field`.

The only thing I can think of as disadvantage is that 3rd parties (e.g. language server, syntax highlighter, linter, ...) have a harder job reimplementing sdoc validation. They get for free what's written as TextX. The rules written in Python are harder to communicate (our perspective) and to implement (their perspective).

Fix patch:
diff --git a/strictdoc/backend/sdoc/grammar/grammar.py b/strictdoc/backend/sdoc/grammar/grammar.py
index c7827eab2..201b7265b 100644
--- a/strictdoc/backend/sdoc/grammar/grammar.py
+++ b/strictdoc/backend/sdoc/grammar/grammar.py
@@ -1,4 +1,4 @@
-REGEX_UID = r"([\w]+[\w\-. ]*)"
+REGEX_UID = r"([\w]+[\w()\-. ]*)"
 
 NEGATIVE_MULTILINE_STRING_START = "(?!>>>\n)"
 NEGATIVE_MULTILINE_STRING_END = "(?!^<<<)"


Test patch:
diff --git a/tests/unit/strictdoc/backend/sdoc/test_dsl_passthrough.py b/tests/unit/strictdoc/backend/sdoc/test_dsl_passthrough.py
index 95c048f68..95055afd5 100644
--- a/tests/unit/strictdoc/backend/sdoc/test_dsl_passthrough.py
+++ b/tests/unit/strictdoc/backend/sdoc/test_dsl_passthrough.py
@@ -217,13 +217,13 @@ def test_022_requirement_uid_and_link():
 TITLE: Test Doc
 
 [REQUIREMENT]
-UID: With spaces and _ and - and . and non latin 特点
+UID: With spaces and (_-.) and non latin 特点
 STATEMENT: >>>
 This is a statement.
 
 [TEXT]
 STATEMENT: >>>
-[LINK: With spaces and _ and - and . and non latin 特点]
+[LINK: With spaces and (_-.) and non latin 特点]
 <<<
 """.lstrip()
 
@@ -976,7 +976,7 @@ def test_edge_case_03_uid_present_but_empty_with_space_character():
         _ = reader.read(input_sdoc)
 
     assert exc_info.type is TextXSyntaxError
-    assert "Expected '([\\w]+[\\w\\-. ]*)'" in exc_info.value.args[0].decode(
+    assert "Expected '([\\w]+[\\w()\\-. ]*)'" in exc_info.value.args[0].decode(
         "utf-8"
     )
 
@@ -997,7 +997,7 @@ def test_edge_case_04_uid_present_but_empty_with_two_space_characters():
         _ = reader.read(input_sdoc)
 
     assert exc_info.type is TextXSyntaxError
-    assert "Expected '([\\w]+[\\w\\-. ]*)'" in exc_info.value.args[0].decode(
+    assert "Expected '([\\w]+[\\w()\\-. ]*)'" in exc_info.value.args[0].decode(
         "utf-8"
     )
 
@@ -1021,8 +1021,8 @@ def test_edge_case_10_empty_multiline_field():
     assert (
         "Expected '^\\[ANCHOR: ' or '[LINK: ' or "
         "'(?ms)(?!^<<<)(?!^\\[\\/FREETEXT\\]\\n)(?!\\[LINK: "
-        "([\\w]+[\\w\\-. ]*))(?!^\\[ANCHOR: "
-        "([\\w]+[\\w\\-. ]*)).'" in exc_info.value.args[0].decode("utf-8")
+        "([\\w]+[\\w()\\-. ]*))(?!^\\[ANCHOR: "
+        "([\\w]+[\\w()\\-. ]*)).'" in exc_info.value.args[0].decode("utf-8")
     )