Repository: strictdoc-project/strictdoc (language: python)

Issue:
[LINK: <uid>] does not allow to express all valid UIDs
Currently there are two different sets of allowed characters for a UID, depending on the UID context:
```
[DOCUMENT]
TITLE: Document UIDs must not contain dots or whitespaces.
UID: FOO_BAR_1

[REQUIREMENT]
UID: FOO BAR 2.
TITLE: Requirement UIDs may contain dots or whitespaces.
STATEMENT: Test.
```

That's because DOCUMENT, SECTION, LINK, ANCHOR syntax use `REGEX_UID = r"([A-Za-z0-9]+[A-Za-z0-9_\-]*)"` in their grammar, while requirements define UID as SDocNodeField which most often is matched with regex `\S.*`. [StrictDoc User Guide - UID](https://strictdoc.readthedocs.io/en/stable/strictdoc_01_user_guide.html#uid) confirms the latter:

> StrictDoc does not impose any limitations on the format of a UID. Examples: ... SAVOIR.OBC.PM.80

As a consequence, one can define `FOO BAR 2.`, but will get an exception when trying to reference it with `[LINK: FOO BAR 2.]`.

IMO:
- any UID that can be defined should also be usable in LINKs
- allowed UID characters should be the same in any context, for consistency and comprehensibility
- allowing whitespaces and dots is useful when copying requirements from external documents/standards to sdoc


Discussion / hints:
You are catching many things that I wanted to do myself for a long time, these are great reminders :+1: 

Your IMOs are also how I see it.
Do you think you could create a good regex to catch all the cases and reuse it everywhere in the grammar? A few unit tests in the `passthrough*` group would be nice to have. See https://github.com/strictdoc-project/strictdoc/blob/main/tests/unit/strictdoc/backend/sdoc/test_dsl_passthrough.py.
> Do you think you could create a good regex to catch all the cases and reuse it everywhere in the grammar?

Yes, should be doable, just can't promise when I get to it.
My initial thought was to have a special line for UID, just as it's already done for MID, something like
```
SDocNode[noskipws]:
  '[' !'SECTION' !SDocCompositeNodeTagName requirement_type = RequirementType ']' '\n'
  ('MID: ' mid = SingleLineString '\n')?
  ('UID: ' uid = UidString '\n')?
  fields *= SDocNodeField
  (
    'RELATIONS:' '\n'
    (relations += Reference)
  )?
;
```

But this would have a similar surprising effect as MID has right know:

With the current release 0.0.57, you can define two `MID` fields for the same element. The first is actually parsed and used as MID, the 2nd one is parsed as ordinary field with field name "MID".

```
[DOCUMENT]
MID: ab8b923ad66a4fa880d7e4892e5a9783
TITLE: How to confuse MIDs.
OPTIONS:
  ENABLE_MID: True

[GRAMMAR]
ELEMENTS:
- TAG: REQUIREMENT
  FIELDS:
  - TITLE: UID
    TYPE: String
    REQUIRED: True
  - TITLE: MID
    TYPE: String
    REQUIRED: False
  - TITLE: STATEMENT
    TYPE: String
    REQUIRED: True

[REQUIREMENT]
MID: cfabae931ae645708bcb95e99ed as77a254a
UID: REQ-1
MID: This is actually not the MID, but an ordinary field.
STATEMENT: Two fields named MID, yet this document validates.
```

If we do the same for UID, TextX will call either `SDocNode.__init__(uid="REQ-1", ...)` or `SDocNode.__init__(fields=[(field_name = "UID", parts = ["REQ-1"],], ...)`, depending on the position of the UID line. The first call would get a regex validated string, the second gets a wildcard string as always for fields.

The least surprising solution is IMO to forbid "MID" and "UID" as SDocNodeField field_name by grammar. What's your opinion?

EDIT: In other words, is it ok to introduce yet another convention where MID/UID must be defined as 1st/2nd fields? Or do we want to allow MID/UID in any position before the content field (which should also be possible but requires different grammar approach)?

Fix patch:
diff --git a/strictdoc/backend/sdoc/grammar/grammar.py b/strictdoc/backend/sdoc/grammar/grammar.py
index 02ff404b2..ba9fe747d 100644
--- a/strictdoc/backend/sdoc/grammar/grammar.py
+++ b/strictdoc/backend/sdoc/grammar/grammar.py
@@ -1,8 +1,9 @@
-REGEX_UID = r"([A-Za-z0-9]+[A-Za-z0-9_\-]*)"
+REGEX_UID = r"([\w]+[\w\-. ]*)"
 
 NEGATIVE_MULTILINE_STRING_START = "(?!>>>\n)"
 NEGATIVE_MULTILINE_STRING_END = "(?!^<<<)"
 NEGATIVE_RELATIONS = "(?!^RELATIONS)"
+NEGATIVE_UID = "(?!^UID)"
 NEGATIVE_FREETEXT_END = "(?!^\\[\\/FREETEXT\\]\n)"
 NEGATIVE_INLINE_LINK_START = rf"(?!\[LINK: {REGEX_UID})"
 NEGATIVE_ANCHOR_START = rf"(?!^\[ANCHOR: {REGEX_UID})"
@@ -44,7 +45,7 @@
 ;
 
 FieldName[noskipws]:
-  /{NEGATIVE_RELATIONS}[A-Z]+[A-Z_]*/
+  /{NEGATIVE_UID}{NEGATIVE_RELATIONS}[A-Z]+[A-Z_]*/
 ;
 """
 
@@ -188,6 +189,8 @@
 
 SDocNodeField[noskipws]:
   (
+    field_name = 'UID' ': ' parts+=/{REGEX_UID}/ '\n'
+    |
     field_name = FieldName ':'
     (
         (' ' parts+=SingleLineTextPart '\n')


Test patch:
diff --git a/tests/unit/strictdoc/backend/sdoc/test_dsl_passthrough.py b/tests/unit/strictdoc/backend/sdoc/test_dsl_passthrough.py
index 57b507a70..535057843 100644
--- a/tests/unit/strictdoc/backend/sdoc/test_dsl_passthrough.py
+++ b/tests/unit/strictdoc/backend/sdoc/test_dsl_passthrough.py
@@ -193,6 +193,33 @@ def test_021_section_and_document_mid():
     assert input_sdoc == output
 
 
+def test_022_requirement_uid_and_link():
+    input_sdoc = """
+[DOCUMENT]
+TITLE: Test Doc
+
+[REQUIREMENT]
+UID: With spaces and _ and - and . and non latin 特点
+STATEMENT: >>>
+This is a statement.
+
+[TEXT]
+STATEMENT: >>>
+[LINK: With spaces and _ and - and . and non latin 特点]
+<<<
+""".lstrip()
+
+    reader = SDReader()
+
+    document = reader.read(input_sdoc)
+    assert isinstance(document, SDocDocument)
+
+    writer = SDWriter()
+    output = writer.write(document)
+
+    assert input_sdoc == output
+
+
 def test_030_multiline_statement():
     input_sdoc = """
 [DOCUMENT]
@@ -912,7 +939,7 @@ def test_edge_case_02_uid_present_but_empty_with_no_space_character():
         _ = reader.read(input_sdoc)
 
     assert exc_info.type is TextXSyntaxError
-    assert "Expected ' '" == exc_info.value.args[0].decode("utf-8")
+    assert "Expected ': '" == exc_info.value.args[0].decode("utf-8")
 
 
 def test_edge_case_03_uid_present_but_empty_with_space_character():
@@ -931,9 +958,8 @@ def test_edge_case_03_uid_present_but_empty_with_space_character():
         _ = reader.read(input_sdoc)
 
     assert exc_info.type is TextXSyntaxError
-    assert (
-        "Expected '^\\[ANCHOR: ' or '[LINK: ' or '(?!>>>\\n)\\S.*' or '>>>\\n'"
-        in exc_info.value.args[0].decode("utf-8")
+    assert "Expected '([\\w]+[\\w\\-. ]*)'" in exc_info.value.args[0].decode(
+        "utf-8"
     )
 
 
@@ -953,9 +979,8 @@ def test_edge_case_04_uid_present_but_empty_with_two_space_characters():
         _ = reader.read(input_sdoc)
 
     assert exc_info.type is TextXSyntaxError
-    assert (
-        "Expected '^\\[ANCHOR: ' or '[LINK: ' or '(?!>>>\\n)\\S.*' or '>>>\\n'"
-        in exc_info.value.args[0].decode("utf-8")
+    assert "Expected '([\\w]+[\\w\\-. ]*)'" in exc_info.value.args[0].decode(
+        "utf-8"
     )
 
 
@@ -978,9 +1003,8 @@ def test_edge_case_10_empty_multiline_field():
     assert (
         "Expected '^\\[ANCHOR: ' or '[LINK: ' or "
         "'(?ms)(?!^<<<)(?!^\\[\\/FREETEXT\\]\\n)(?!\\[LINK: "
-        "([A-Za-z0-9]+[A-Za-z0-9_\\-]*))(?!^\\[ANCHOR: "
-        "([A-Za-z0-9]+[A-Za-z0-9_\\-]*)).'"
-        in exc_info.value.args[0].decode("utf-8")
+        "([\\w]+[\\w\\-. ]*))(?!^\\[ANCHOR: "
+        "([\\w]+[\\w\\-. ]*)).'" in exc_info.value.args[0].decode("utf-8")
     )