Repository: keymanapp/keyman
Pull request #13532: fix(developer): add validation of `nul` usage in output part of rule
Author: login: mcdurdin
Labels: name: developer/
name: developer/compilers/
name: fix
Created: 2025-03-18T05:04:46Z | Merged: 2025-03-18T23:12:55Z

If `nul` is used in the output part of a rule, then text or text-emitting statements cannot also be included in the output of that rule. This was not enforced in earlier versions of Keyman.

Adds a unit test for the enhanced validation.

Fixes: #13455
Relates-to: keymanapp/keyboards#3379

@keymanapp-test-bot skip


Diff:
diff --git a/developer/src/common/include/kmn_compiler_errors.h b/developer/src/common/include/kmn_compiler_errors.h
index c951fab3300..44dc5e33596 100644
--- a/developer/src/common/include/kmn_compiler_errors.h
+++ b/developer/src/common/include/kmn_compiler_errors.h
@@ -275,8 +275,10 @@ namespace KmnCompilerMessages {
     ERROR_ContextExCannotReferenceIf =                  SevError | 0x0B1,
     ERROR_ContextExCannotReferenceNul =                 SevError | 0x0B2,
 
-    FATAL_BufferOverflow =                                SevFatal | 0x0C0
-//    FATAL_Break =                                         SevFatal | 0x0C1,      unused
+    ERROR_TextBeforeOrAfterNulInOutput =                SevError | 0x0B3,
+
+    FATAL_BufferOverflow =                              SevFatal | 0x0C0
+//    FATAL_Break =                                       SevFatal | 0x0C1,      unused
   };
 }
 
diff --git a/developer/src/kmc-kmn/src/compiler/kmn-compiler-messages.ts b/developer/src/kmc-kmn/src/compiler/kmn-compiler-messages.ts
index 9520fe7dcda..e35d4afb44c 100644
--- a/developer/src/kmc-kmn/src/compiler/kmn-compiler-messages.ts
+++ b/developer/src/kmc-kmn/src/compiler/kmn-compiler-messages.ts
@@ -747,11 +747,30 @@ export class KmnCompilerMessages {
   );
 
   static ERROR_ContextExCannotReferenceNul                    = SevError | 0x0B2;
-  static Error_ContextExCannotReferenceNul                     = () => m(
+  static Error_ContextExCannotReferenceNul                    = () => m(
     this.ERROR_ContextExCannotReferenceNul,
     `The offset in context() points to a non-character nul statement in the context`,
   );
 
+  static ERROR_TextBeforeOrAfterNulInOutput                   = SevError | 0x0B3;
+  static Error_TextBeforeOrAfterNulInOutput                    = () => m(
+    this.ERROR_TextBeforeOrAfterNulInOutput,
+    `A rule with a nul statement in the output cannot also output text or text-emitting statements`,
+    `If a rule has [\`nul\`](https://help.keyman.com/developer/language/reference/nul)
+    in the output, it means that it deletes any context referenced on the left-hand
+    side of the rule, and has no other output, so it does not make sense to have any
+    content apart from the \`nul\` statement in the output.
+
+    The following statements can still be used with \`nul\` in the output:
+    * [\`beep()\`](https://help.keyman.com/developer/language/reference/beep)
+    * [\`call()\`](https://help.keyman.com/developer/language/reference/call)
+    * [\`reset()\`](https://help.keyman.com/developer/language/reference/reset)
+    * [\`return\`](https://help.keyman.com/developer/language/reference/return)
+    * [\`save()\`](https://help.keyman.com/developer/language/reference/save)
+    * [\`set()\`](https://help.keyman.com/developer/language/reference/set)
+    * [\`use()\`](https://help.keyman.com/developer/language/reference/use)`
+  );
+
   static FATAL_BufferOverflow                                 = SevFatal | 0x0C0;
   static Fatal_BufferOverflow                                 = () => m(this.FATAL_BufferOverflow, `The compiler memory buffer overflowed`);
 
diff --git a/developer/src/kmc-kmn/test/fixtures/invalid-keyboards/error_text_before_or_after_nul_in_output_after.kmn b/developer/src/kmc-kmn/test/fixtures/invalid-keyboards/error_text_before_or_after_nul_in_output_after.kmn
new file mode 100644
index 00000000000..1ca1463c73b
--- /dev/null
+++ b/developer/src/kmc-kmn/test/fixtures/invalid-keyboards/error_text_before_or_after_nul_in_output_after.kmn
@@ -0,0 +1,9 @@
+store(&NAME) 'ERROR_TextBeforeOrAfterNulInOutput'
+store(&VERSION) '9.0'
+
+begin Unicode > use(main)
+
+group(main) using keys
+
+c ERROR_TextBeforeOrAfterNulInOutput after nul
++ 'a' > nul 'a'
diff --git a/developer/src/kmc-kmn/test/fixtures/invalid-keyboards/error_text_before_or_after_nul_in_output_before.kmn b/developer/src/kmc-kmn/test/fixtures/invalid-keyboards/error_text_before_or_after_nul_in_output_before.kmn
new file mode 100644
index 00000000000..bcb8f5c089e
--- /dev/null
+++ b/developer/src/kmc-kmn/test/fixtures/invalid-keyboards/error_text_before_or_after_nul_in_output_before.kmn
@@ -0,0 +1,9 @@
+store(&NAME) 'ERROR_TextBeforeOrAfterNulInOutput'
+store(&VERSION) '9.0'
+
+begin Unicode > use(main)
+
+group(main) using keys
+
+c ERROR_TextBeforeOrAfterNulInOutput before nul
++ 'b' > 'b' nul
diff --git a/developer/src/kmc-kmn/test/messages.tests.ts b/developer/src/kmc-kmn/test/kmn-compiler-messages.tests.ts
similarity index 95%
rename from developer/src/kmc-kmn/test/messages.tests.ts
rename to developer/src/kmc-kmn/test/kmn-compiler-messages.tests.ts
index 16c9ab61500..74b471c36a1 100644
--- a/developer/src/kmc-kmn/test/messages.tests.ts
+++ b/developer/src/kmc-kmn/test/kmn-compiler-messages.tests.ts
@@ -1,10 +1,10 @@
 import 'mocha';
 import { assert } from 'chai';
-import { KmnCompilerMessageRanges, KmnCompilerMessages } from '../src/compiler/kmn-compiler-messages.js';
+import { CompilerErrorMask, CompilerErrorNamespace } from '@keymanapp/developer-utils';
 import { TestCompilerCallbacks, verifyCompilerMessagesObject } from '@keymanapp/developer-test-helpers';
-import { makePathToFixture } from './helpers/index.js';
 import { KmnCompiler } from '../src/main.js';
-import { CompilerErrorMask, CompilerErrorNamespace } from '@keymanapp/developer-utils';
+import { KmnCompilerMessageRanges, KmnCompilerMessages } from '../src/compiler/kmn-compiler-messages.js';
+import { makePathToFixture } from './helpers/index.js';
 
 describe('KmnCompilerMessages', function () {
   const callbacks = new TestCompilerCallbacks();
@@ -216,4 +216,11 @@ describe('KmnCompilerMessages', function () {
     await testForMessage(this, ['invalid-keyboards', 'error_contextex_cannot_reference_nul.kmn'], KmnCompilerMessages.ERROR_ContextExCannotReferenceNul);
   });
 
+  // ERROR_TextBeforeOrAfterNulInOutput
+
+  it('should generate ERROR_TextBeforeOrAfterNulInOutput if text before or after nul in output', async function() {
+    await testForMessage(this, ['invalid-keyboards', 'error_text_before_or_after_nul_in_output_before.kmn'], KmnCompilerMessages.ERROR_TextBeforeOrAfterNulInOutput);
+    await testForMessage(this, ['invalid-keyboards', 'error_text_before_or_after_nul_in_output_after.kmn'], KmnCompilerMessages.ERROR_TextBeforeOrAfterNulInOutput);
+  });
+
 });
diff --git a/developer/src/kmcmplib/src/Compiler.cpp b/developer/src/kmcmplib/src/Compiler.cpp
index 4774f78380b..3b53453dceb 100644
--- a/developer/src/kmcmplib/src/Compiler.cpp
+++ b/developer/src/kmcmplib/src/Compiler.cpp
@@ -1556,6 +1556,71 @@ KMX_DWORD CheckVirtualKeysInOutput(PKMX_WCHAR output) {
   return STATUS_Success;
 }
 
+const KMX_BOOL CODE__IS_TEXTUAL[] = {
+  -1,     // undefined                0x00
+  TRUE,   // CODE_ANY                 0x01
+  TRUE,   // CODE_INDEX               0x02
+  TRUE,   // CODE_CONTEXT             0x03
+  FALSE,  // CODE_NUL                 0x04
+  FALSE,  // CODE_USE                 0x05 // may trigger text effects but indirectly
+  FALSE,  // CODE_RETURN              0x06
+  FALSE,  // CODE_BEEP                0x07
+  TRUE,   // CODE_DEADKEY             0x08
+  -1,     // unused                   0x09
+  TRUE,   // CODE_EXTENDED            0x0A
+  -1,     // CODE_EXTENDEDEND         0x0B (unused)
+  FALSE,  // CODE_SWITCH              0x0C
+  -1,     // CODE_KEY                 0x0D (never used)
+  FALSE,  // CODE_CLEARCONTEXT        0x0E
+  FALSE,  // CODE_CALL                0x0F // may trigger text effects but indirectly
+  -1,     // UC_SENTINEL_EXTENDEDEND  0x10 (not valid with UC_SENTINEL)
+  TRUE,   // CODE_CONTEXTEX           0x11
+  TRUE,   // CODE_NOTANY              0x12
+  FALSE,  // CODE_SETOPT              0x13
+  FALSE,  // CODE_IFOPT               0x14
+  FALSE,  // CODE_SAVEOPT             0x15
+  FALSE,  // CODE_RESETOPT            0x16
+  FALSE,  // CODE_IFSYSTEMSTORE       0x17
+  FALSE   // CODE_SETSYSTEMSTORE      0x18
+};
+
+static_assert(sizeof(CODE__IS_TEXTUAL) / sizeof(CODE__IS_TEXTUAL[0]) == (CODE_LASTCODE + 1), "Size of array CODE__EMITS_TEXT not correct");
+
+KMX_BOOL IsTextualCode(KMX_WORD code) {
+  assert(code >= CODE_ANY && code <= CODE_LASTCODE);
+  if(!(code >= CODE_ANY && code <= CODE_LASTCODE)) {
+    return FALSE;
+  }
+  return CODE__IS_TEXTUAL[code];
+}
+
+/**
+ * Warn if output has text before or after a nul statement, which
+ * doesn't make sense -- if nul is used, it indicates no output
+ */
+KMX_DWORD CheckNulUsageInOutput(PKMX_WCHAR output) {
+  PKMX_WCHAR p;
+  KMX_BOOL hasNul = FALSE, hasOther = FALSE;
+  for (p = output; *p; p = incxstr(p)) {
+    if (*p == UC_SENTINEL) {
+      auto code = *(p+1);
+      if(code == CODE_NUL) {
+        hasNul = TRUE;
+      } else if(IsTextualCode(code)) {
+        hasOther = TRUE;
+      }
+    } else {
+      hasOther = TRUE;
+    }
+  }
+
+  if(hasNul && hasOther) {
+    return KmnCompilerMessages::ERROR_TextBeforeOrAfterNulInOutput;
+  }
+
+  return STATUS_Success;
+}
+
 /**
  * Adds implicit `context` to start of output of rules for readonly groups
  */
@@ -1694,6 +1759,10 @@ KMX_DWORD ProcessKeyLineImpl(PFILE_KEYBOARD fk, PKMX_WCHAR str, KMX_BOOL IsUnico
     return msg;
   }
 
+  if ((msg = CheckNulUsageInOutput(pklOut)) != STATUS_Success) {
+    return msg;
+  }
+
   if (gp->fReadOnly) {
     // Ensure no output is made from the rule, and that
     // use() statements meet required readonly semantics
diff --git a/developer/src/kmcmplib/tests/fixtures/keyboards-repo/mozhi_malayalam.kmx b/developer/src/kmcmplib/tests/fixtures/keyboards-repo/mozhi_malayalam.kmx
deleted file mode 100644
index 1c0137b88ca..00000000000
Binary files a/developer/src/kmcmplib/tests/fixtures/keyboards-repo/mozhi_malayalam.kmx and /dev/null differ


Reviews:
user: login: mcdurdin
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-18T05:07:44Z
user: login: srl295
state: APPROVED
author_association: MEMBER
submitted_at: 2025-03-18T17:01:18Z
user: login: mcdurdin
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-18T23:12:14Z

Review comments:
diff_hunk: @@ -1556,6 +1556,71 @@ KMX_DWORD CheckVirtualKeysInOutput(PKMX_WCHAR output) {
   return STATUS_Success;
 }
 
+const KMX_BOOL CODE__IS_TEXTUAL[] = {
+  -1,     // undefined                0x00
+  TRUE,   // CODE_ANY                 0x01
+  TRUE,   // CODE_INDEX               0x02
+  TRUE,   // CODE_CONTEXT             0x03
+  FALSE,  // CODE_NUL                 0x04
+  FALSE,  // CODE_USE                 0x05 // may trigger text effects but indirectly
+  FALSE,  // CODE_RETURN              0x06
+  FALSE,  // CODE_BEEP                0x07
+  TRUE,   // CODE_DEADKEY             0x08
+  -1,     // unused                   0x09
+  TRUE,   // CODE_EXTENDED            0x0A
+  -1,     // CODE_EXTENDEDEND         0x0B (unused)
+  FALSE,  // CODE_SWITCH              0x0C
+  -1,     // CODE_KEY                 0x0D (never used)
+  FALSE,  // CODE_CLEARCONTEXT        0x0E
+  FALSE,  // CODE_CALL                0x0F // may trigger text effects but indirectly
+  -1,     // UC_SENTINEL_EXTENDEDEND  0x10 (not valid with UC_SENTINEL)
+  TRUE,   // CODE_CONTEXTEX           0x11
+  TRUE,   // CODE_NOTANY              0x12
+  FALSE,  // CODE_SETOPT              0x13
+  FALSE,  // CODE_IFOPT               0x14
+  FALSE,  // CODE_SAVEOPT             0x15
+  FALSE,  // CODE_RESETOPT            0x16
+  FALSE,  // CODE_IFSYSTEMSTORE       0x17
+  FALSE   // CODE_SETSYSTEMSTORE      0x18
+};
+
+static_assert(sizeof(CODE__IS_TEXTUAL) / sizeof(CODE__IS_TEXTUAL[0]) == (CODE_LASTCODE + 1), "Size of array CODE__EMITS_TEXT not correct");
path: developer/src/kmcmplib/src/Compiler.cpp
user: login: mcdurdin
body: ```suggestion
static_assert(sizeof(CODE__IS_TEXTUAL) / sizeof(CODE__IS_TEXTUAL[0]) == (CODE_LASTCODE + 1), "Size of array CODE__IS_TEXTUAL not correct");
```
created_at: 2025-03-18T05:07:44Z
author_association: MEMBER
diff_hunk: @@ -1556,6 +1556,71 @@ KMX_DWORD CheckVirtualKeysInOutput(PKMX_WCHAR output) {
   return STATUS_Success;
 }
 
+const KMX_BOOL CODE__IS_TEXTUAL[] = {
+  -1,     // undefined                0x00
+  TRUE,   // CODE_ANY                 0x01
+  TRUE,   // CODE_INDEX               0x02
+  TRUE,   // CODE_CONTEXT             0x03
+  FALSE,  // CODE_NUL                 0x04
+  FALSE,  // CODE_USE                 0x05 // may trigger text effects but indirectly
+  FALSE,  // CODE_RETURN              0x06
+  FALSE,  // CODE_BEEP                0x07
+  TRUE,   // CODE_DEADKEY             0x08
+  -1,     // unused                   0x09
+  TRUE,   // CODE_EXTENDED            0x0A
+  -1,     // CODE_EXTENDEDEND         0x0B (unused)
+  FALSE,  // CODE_SWITCH              0x0C
+  -1,     // CODE_KEY                 0x0D (never used)
+  FALSE,  // CODE_CLEARCONTEXT        0x0E
+  FALSE,  // CODE_CALL                0x0F // may trigger text effects but indirectly
+  -1,     // UC_SENTINEL_EXTENDEDEND  0x10 (not valid with UC_SENTINEL)
+  TRUE,   // CODE_CONTEXTEX           0x11
+  TRUE,   // CODE_NOTANY              0x12
+  FALSE,  // CODE_SETOPT              0x13
+  FALSE,  // CODE_IFOPT               0x14
+  FALSE,  // CODE_SAVEOPT             0x15
+  FALSE,  // CODE_RESETOPT            0x16
+  FALSE,  // CODE_IFSYSTEMSTORE       0x17
+  FALSE   // CODE_SETSYSTEMSTORE      0x18
+};
+
+static_assert(sizeof(CODE__IS_TEXTUAL) / sizeof(CODE__IS_TEXTUAL[0]) == (CODE_LASTCODE + 1), "Size of array CODE__EMITS_TEXT not correct");
+
+KMX_BOOL IsTextualCode(KMX_WORD code) {
+  assert(code >= CODE_ANY && code <= CODE_LASTCODE);
+  if(!(code >= CODE_ANY && code <= CODE_LASTCODE)) {
+    return FALSE;
+  }
+  return CODE__IS_TEXTUAL[code];
+}
+
+/**
+ * Warn if output has text before or after a nul statement, which
+ * doesn't make sense -- if nul is used, it indicates no output
+ */
+KMX_DWORD CheckNulUsageInOutput(PKMX_WCHAR output) {
+  PKMX_WCHAR p;
+  KMX_BOOL hasNul = FALSE, hasOther = FALSE;
+  for (p = output; *p; p = incxstr(p)) {
+    if (*p == UC_SENTINEL) {
+      auto code = *(p+1);
+      if(code == CODE_NUL) {
path: developer/src/kmcmplib/src/Compiler.cpp
user: login: srl295
body: side note - interesting to see other use of UC_SENTINEL
created_at: 2025-03-18T17:01:07Z
author_association: MEMBER
diff_hunk: @@ -1556,6 +1556,71 @@ KMX_DWORD CheckVirtualKeysInOutput(PKMX_WCHAR output) {
   return STATUS_Success;
 }
 
+const KMX_BOOL CODE__IS_TEXTUAL[] = {
+  -1,     // undefined                0x00
+  TRUE,   // CODE_ANY                 0x01
+  TRUE,   // CODE_INDEX               0x02
+  TRUE,   // CODE_CONTEXT             0x03
+  FALSE,  // CODE_NUL                 0x04
+  FALSE,  // CODE_USE                 0x05 // may trigger text effects but indirectly
+  FALSE,  // CODE_RETURN              0x06
+  FALSE,  // CODE_BEEP                0x07
+  TRUE,   // CODE_DEADKEY             0x08
+  -1,     // unused                   0x09
+  TRUE,   // CODE_EXTENDED            0x0A
+  -1,     // CODE_EXTENDEDEND         0x0B (unused)
+  FALSE,  // CODE_SWITCH              0x0C
+  -1,     // CODE_KEY                 0x0D (never used)
+  FALSE,  // CODE_CLEARCONTEXT        0x0E
+  FALSE,  // CODE_CALL                0x0F // may trigger text effects but indirectly
+  -1,     // UC_SENTINEL_EXTENDEDEND  0x10 (not valid with UC_SENTINEL)
+  TRUE,   // CODE_CONTEXTEX           0x11
+  TRUE,   // CODE_NOTANY              0x12
+  FALSE,  // CODE_SETOPT              0x13
+  FALSE,  // CODE_IFOPT               0x14
+  FALSE,  // CODE_SAVEOPT             0x15
+  FALSE,  // CODE_RESETOPT            0x16
+  FALSE,  // CODE_IFSYSTEMSTORE       0x17
+  FALSE   // CODE_SETSYSTEMSTORE      0x18
+};
+
+static_assert(sizeof(CODE__IS_TEXTUAL) / sizeof(CODE__IS_TEXTUAL[0]) == (CODE_LASTCODE + 1), "Size of array CODE__EMITS_TEXT not correct");
+
+KMX_BOOL IsTextualCode(KMX_WORD code) {
+  assert(code >= CODE_ANY && code <= CODE_LASTCODE);
+  if(!(code >= CODE_ANY && code <= CODE_LASTCODE)) {
+    return FALSE;
+  }
+  return CODE__IS_TEXTUAL[code];
+}
+
+/**
+ * Warn if output has text before or after a nul statement, which
+ * doesn't make sense -- if nul is used, it indicates no output
+ */
+KMX_DWORD CheckNulUsageInOutput(PKMX_WCHAR output) {
+  PKMX_WCHAR p;
+  KMX_BOOL hasNul = FALSE, hasOther = FALSE;
+  for (p = output; *p; p = incxstr(p)) {
+    if (*p == UC_SENTINEL) {
+      auto code = *(p+1);
+      if(code == CODE_NUL) {
path: developer/src/kmcmplib/src/Compiler.cpp
user: login: mcdurdin
body: `UC_SENTINEL` is the flagship character of .kmx. Take a look at `incxstr` and its cousins for all the joy!
created_at: 2025-03-18T23:12:14Z
author_association: MEMBER

PR comments:
user: login: keymanapp-test-bot[bot]
created_at: 2025-03-18T05:04:50Z
body: # User Test Results

[Test specification and instructions](https://github.com/keymanapp/keyman/pull/13532#)

_User tests are not required_

## Test Artifacts

* **Developer**
  * [**Keyman Developer**](https://build.palaso.org/repository/download/Keyman_Developer_Test/532670:id/release/18.0.216/keymandeveloper-18.0.216.exe)
  * [Compiler Regression Tests](https://build.palaso.org/repository/download/Keyman_Developer_Test/532670:id/keyboard-regression/)
  * [kmcomp.zip](https://build.palaso.org/repository/download/Keyman_Developer_Test/532670:id/release/18.0.216/kmcomp-18.0.216.zip)
* **Keyboards**
  * [Test Keyboards](https://build.palaso.org/repository/download/Keyman_Developer_Test/532670:id/keyboards/index.html)
author_association: NONE
user: login: mcdurdin
created_at: 2025-03-18T05:15:08Z
body: Failing Developer builds: as mozhi_malayalam has rules that do not meet this new validation requirement (see keymanapp/keyboards#3379), I need to remove it from the baseline test in kmcmplib.

```
562/701 mozhi_malayalam                         FAIL            0.09s   exit status 108
```
author_association: MEMBER
user: login: mcdurdin
created_at: 2025-03-18T06:53:14Z
body: (re-ready for review)
author_association: MEMBER
user: login: keyman-server
created_at: 2025-03-19T18:03:13Z
body: Changes in this pull request will be available for download in [Keyman version 18.0.218-beta](https://keyman.com/downloads/releases/beta/18.0.218)
author_association: COLLABORATOR