Repository: PyThaiNLP/pythainlp (python)

Issue:
bug: Why isn’t space preprocessing consistent between Longest Matching and Multi-Cut?
### Description

Hi,

I noticed that Multi-Cut preprocesses spaces (e.g., grouping consecutive spaces into one token), while Longest Matching does not. Why not preprocess spaces the same way for both tokenizers to ensure consistency?

Thanks for your clarification!

### Expected results

A clear explanation

### Current results

Multi-Cut preprocesses spaces (e.g., grouping consecutive spaces into one token), while Longest Matching does not.

### Steps to reproduce

-

### PyThaiNLP version

5.0.5

### Python version

3.9.6

### Operating system and version

Google Colab Latest

### More info

_No response_

### Possible solution

_No response_

### Files

_No response_

Pull request description:
Fix bug in Longest Matching tokenizer to preprocess spaces consistently
Fixes #1061

Update the Longest Matching tokenizer to preprocess spaces consistently with the Multi-Cut tokenizer.

* Modify `pythainlp/tokenize/longest.py` to group consecutive spaces into one token using regex.
* Add test cases in `tests/core/test_tokenize.py` to verify consistent preprocessing of spaces between Longest Matching and Multi-Cut tokenizers.


Patch:
diff --git a/pythainlp/tokenize/longest.py b/pythainlp/tokenize/longest.py
index 8bd21bf4..1b0bf718 100644
--- a/pythainlp/tokenize/longest.py
+++ b/pythainlp/tokenize/longest.py
@@ -38,6 +38,7 @@ _REAR_DEP_CHAR = ["ั", "ื", "เ", "แ", "โ", "ใ", "ไ", "ํ"]
 _TRAILING_CHAR = ["ๆ", "ฯ"]
 
 _RE_NONTHAI = re.compile(r"[A-Za-z\d]*")
+_RE_SPACES = re.compile(r"\s+")
 
 _KNOWN = True
 _UNKNOWN = False
@@ -134,7 +135,15 @@ class LongestMatchTokenizer:
                     token_statuses.append(_KNOWN)
                 begin_pos += len(match)
 
-        return tokens
+        # Group consecutive spaces into one token
+        grouped_tokens = []
+        for token in tokens:
+            if token.isspace() and grouped_tokens and grouped_tokens[-1].isspace():
+                grouped_tokens[-1] += token
+            else:
+                grouped_tokens.append(token)
+
+        return grouped_tokens
 
     def tokenize(self, text: str) -> List[str]:
         tokens = self.__segment(text)


Test patch:
diff --git a/tests/core/test_tokenize.py b/tests/core/test_tokenize.py
index 054a2ab7..c1fd7ba0 100644
--- a/tests/core/test_tokenize.py
+++ b/tests/core/test_tokenize.py
@@ -390,6 +390,18 @@ class TokenizeTestCase(unittest.TestCase):
             longest_tokenizer.word_tokenize("เฉียบพลัน"),
             ["เฉียบพลัน"],
         )
+        self.assertEqual(
+            longest.segment("ทดสอบ  ทดสอบ  ทดสอบ"),
+            ["ทดสอบ", "  ", "ทดสอบ", "  ", "ทดสอบ"],
+        )
+        self.assertEqual(
+            longest.segment("ทดสอบ  ทดสอบ"),
+            ["ทดสอบ", "  ", "ทดสอบ"],
+        )
+        self.assertEqual(
+            longest.segment("ทดสอบ    ทดสอบ"),
+            ["ทดสอบ", "    ", "ทดสอบ"],
+        )
 
     def test_mm(self):
         self.assertEqual(multi_cut.segment(None), [])