Repository: checkstyle/checkstyle

Problem statement:
Version 9.x reports NoWhitespaceBefore error on semicolon or comma after string with emoji
I have read check documentation: https://checkstyle.org/config_whitespace.html#NoWhitespaceBefore
I have downloaded the latest checkstyle from: https://checkstyle.org/cmdline.html#Download_and_Run
I have executed the cli and showed it below, as cli describes the problem better than 1,000 words

```bash
/var/tmp $ javac Test.java

/var/tmp $ cat config.xml
<?xml version="1.0"?>
<!DOCTYPE module PUBLIC
        "-//Checkstyle//DTD Checkstyle Configuration 1.3//EN"
        "https://checkstyle.org/dtds/configuration_1_3.dtd">
<module name="Checker">
    <module name="TreeWalker">
        <module name="NoWhitespaceBefore"/>
    </module>
</module>

/var/tmp $ cat Test.java
public class Test {
    private String[] _mVar0 = {
        "😃😉🙈" ,  // should be error
        "😃 😉 🙈" ,  // should be error
        "😃 😉 🙈",  // should ok
        "😃😉🙈"  // should be ok
    };
    private String _mVar1 = "😃😉🙈"; // should be ok
    private String _mVar2 = "😃 😉 🙈"; // should be ok
    private String _mVar3 = "😃 😉 🙈" ; // should be error
    private String _mVar4 = "a b c" ; // should be error
}

/var/tmp $ RUN_LOCALE="-Duser.language=en -Duser.country=US"
/var/tmp $ java $RUN_LOCALE -jar checkstyle-8.45.1-all.jar -c config.xml Test.java
Starting audit...
[ERROR] /home/jogchem/Projects/checkstyle/Test.java:3:18: ',' is preceded with whitespace. [NoWhitespaceBefore]
[ERROR] /home/jogchem/Projects/checkstyle/Test.java:4:20: ',' is preceded with whitespace. [NoWhitespaceBefore]
[ERROR] /home/jogchem/Projects/checkstyle/Test.java:10:40: ';' is preceded with whitespace. [NoWhitespaceBefore]
[ERROR] /home/jogchem/Projects/checkstyle/Test.java:11:37: ';' is preceded with whitespace. [NoWhitespaceBefore]
Audit done.
Checkstyle ends with 4 errors.

/var/tmp $ java $RUN_LOCALE -jar checkstyle-9.1-all.jar -c config.xml Test.java
Starting audit...
[ERROR] /home/jogchem/Projects/checkstyle/Test.java:5:16: ',' is preceded with whitespace. [NoWhitespaceBefore]
[ERROR] /home/jogchem/Projects/checkstyle/Test.java:9:36: ';' is preceded with whitespace. [NoWhitespaceBefore]
[ERROR] /home/jogchem/Projects/checkstyle/Test.java:11:37: ';' is preceded with whitespace. [NoWhitespaceBefore]
Audit done.
Checkstyle ends with 3 errors.
```

For Windows users, please use `type` instead of `cat` and run

```cmd
set RUN_LOCALE="-Duser.language=en -Duser.country=US"
java %RUN_LOCALE% -jar checkstyle-8.45.1-all.jar -c config.xml Test.java
java %RUN_LOCALE% -jar checkstyle-9.1-all.jar -c config.xml Test.java
```

in place of the last 2 commands above.

---

I expect the output of version 9.1 to be the same as for version 8.45.1, but instead it does not report errors for whitespace that follows a string with emoji. Even weirder is that I discovered this bug because it did report extra errors after upgrading to 9.1, so it seems that running the checkstyle from the maven plugin leads to different behaviour.

I've created a test case to reproduce this as well. I think the root cause of this issue is that ANTLR generates a different offset for the SEMI or COMMA token. Using the test file https://github.com/dijkstraj/checkstyle/blob/emoji-8/src/test/resources/com/puppycrawl/tools/checkstyle/grammar/InputAstRegressionEmojiString.java the generated AST (https://github.com/dijkstraj/checkstyle/blob/emoji-8/src/test/resources/com/puppycrawl/tools/checkstyle/grammar/InputAstRegressionEmojiString.txt) has columns 33 and 34 for the semicolons in version 8.45.1, but columns 31 and 34 in master (https://github.com/dijkstraj/checkstyle/blob/emoji-9/src/test/resources/com/puppycrawl/tools/checkstyle/grammar/InputAstRegressionEmojiString.txt).
So it looks like a difference between character and byte offset.

The error that I originally ran into is reproducable with this line: https://github.com/dijkstraj/checkstyle/blob/emoji-9/src/test/resources/com/puppycrawl/tools/checkstyle/checks/whitespace/nowhitespacebefore/InputNoWhitespaceBeforeDefault.java#L309


Hints:
it looks similar to https://github.com/checkstyle/checkstyle/issues/10837, @nmancus1 can you help?

Patch:
diff --git a/src/main/java/com/puppycrawl/tools/checkstyle/checks/whitespace/NoWhitespaceBeforeCheck.java b/src/main/java/com/puppycrawl/tools/checkstyle/checks/whitespace/NoWhitespaceBeforeCheck.java
--- a/src/main/java/com/puppycrawl/tools/checkstyle/checks/whitespace/NoWhitespaceBeforeCheck.java
+++ b/src/main/java/com/puppycrawl/tools/checkstyle/checks/whitespace/NoWhitespaceBeforeCheck.java
@@ -211,13 +211,14 @@ public int[] getRequiredTokens() {
     public void visitToken(DetailAST ast) {
         final String line = getLine(ast.getLineNo() - 1);
         final int before = ast.getColumnNo() - 1;
+        final int[] codePoints = line.codePoints().toArray();
 
-        if ((before == -1 || Character.isWhitespace(line.charAt(before)))
+        if ((before == -1 || isWhitespace(codePoints, before))
                 && !isInEmptyForInitializerOrCondition(ast)) {
             boolean flag = !allowLineBreaks;
             // verify all characters before '.' are whitespace
             for (int i = 0; i <= before - 1; i++) {
-                if (!Character.isWhitespace(line.charAt(i))) {
+                if (!isWhitespace(codePoints, i)) {
                     flag = true;
                     break;
                 }
@@ -228,6 +229,23 @@ public void visitToken(DetailAST ast) {
         }
     }
 
+    /**
+     * Converts the Unicode code point at index {@code index} to it's UTF-16
+     * representation, then checks if the character is whitespace. Note that the given
+     * index {@code index} should correspond to the location of the character
+     * to check in the string, not in code points.
+     *
+     * @param codePoints the array of Unicode code points
+     * @param index the index of the character to check
+     * @return true if character at {@code index} is whitespace
+     */
+    private static boolean isWhitespace(int[] codePoints, int index) {
+        //  We only need to check the first member of a surrogate pair to verify that
+        //  it is not whitespace.
+        final char character = Character.toChars(codePoints[index])[0];
+        return Character.isWhitespace(character);
+    }
+
     /**
      * Checks that semicolon is in empty for initializer or condition.
      *


Test patch:
diff --git a/src/test/java/com/puppycrawl/tools/checkstyle/checks/whitespace/NoWhitespaceBeforeCheckTest.java b/src/test/java/com/puppycrawl/tools/checkstyle/checks/whitespace/NoWhitespaceBeforeCheckTest.java
--- a/src/test/java/com/puppycrawl/tools/checkstyle/checks/whitespace/NoWhitespaceBeforeCheckTest.java
+++ b/src/test/java/com/puppycrawl/tools/checkstyle/checks/whitespace/NoWhitespaceBeforeCheckTest.java
@@ -127,4 +127,17 @@ public void testNoWhitespaceBeforeTextBlocksWithTabIndent() throws Exception {
                 getNonCompilablePath("InputNoWhitespaceBeforeTextBlocksTabIndent.java"), expected);
     }
 
+    @Test
+    public void testNoWhitespaceBeforeWithEmoji() throws Exception {
+        final String[] expected = {
+            "13:15: " + getCheckMessage(MSG_KEY, ","),
+            "14:17: " + getCheckMessage(MSG_KEY, ","),
+            "20:37: " + getCheckMessage(MSG_KEY, ";"),
+            "21:37: " + getCheckMessage(MSG_KEY, ";"),
+        };
+
+        verifyWithInlineConfigParser(
+                getPath("InputNoWhitespaceBeforeWithEmoji.java"), expected);
+    }
+
 }
diff --git a/src/test/resources/com/puppycrawl/tools/checkstyle/checks/whitespace/nowhitespacebefore/InputNoWhitespaceBeforeWithEmoji.java b/src/test/resources/com/puppycrawl/tools/checkstyle/checks/whitespace/nowhitespacebefore/InputNoWhitespaceBeforeWithEmoji.java
new file mode 100644
--- /dev/null
+++ b/src/test/resources/com/puppycrawl/tools/checkstyle/checks/whitespace/nowhitespacebefore/InputNoWhitespaceBeforeWithEmoji.java
@@ -0,0 +1,22 @@
+/*
+NoWhitespaceBefore
+allowLineBreaks = (default)false
+tokens = (default)COMMA, SEMI, POST_INC, POST_DEC, ELLIPSIS, LABELED_STAT
+
+
+*/
+
+package com.puppycrawl.tools.checkstyle.checks.whitespace.nowhitespacebefore;
+
+public class InputNoWhitespaceBeforeWithEmoji {
+    private String[] _mVar0 = {
+        "😃😉🙈" ,  // violation
+        "😃 😉 🙈" ,  // violation
+        "😃 😉 🙈",  // ok
+        "😃😉🙈"  // ok
+    };
+    private String _mVar1 = "😃😉🙈"; // ok
+    private String _mVar2 = "😃 😉 🙈"; // ok
+    private String _mVar3 = "😃 😉 🙈" ; // violation
+    private String _mVar4 = "a b c" ; // violation
+}