Repository: PyThaiNLP/pythainlp

Issue:
bug: Why isn’t space preprocessing consistent between Longest Matching and Multi-Cut?
### Description

Hi,

I noticed that Multi-Cut preprocesses spaces (e.g., grouping consecutive spaces into one token), while Longest Matching does not. Why not preprocess spaces the same way for both tokenizers to ensure consistency?

Thanks for your clarification!

### Expected results

A clear explanation

### Current results

Multi-Cut preprocesses spaces (e.g., grouping consecutive spaces into one token), while Longest Matching does not.

### Steps to reproduce

-

### PyThaiNLP version

5.0.5

### Python version

3.9.6

### Operating system and version

Google Colab Latest

### More info

_No response_

### Possible solution

_No response_

### Files

_No response_

Discussion:
@github-actions: Hello @Muaykillz, thank you for your interest in our work!

If this is a bug report, please provide screenshots and **minimum viable code to reproduce your issue**, otherwise we can not help you.

สวัสดี @Muaykillz ขอบคุณที่สนใจงานของเรา

ถ้านี่เป็นรายงานข้อผิดพลาด กรุณาแนบภาพหน้าจอ ข้อความแสดงข้อผิดพลาด และ **โค้ดที่สั้นที่สุดเท่าที่จะทำให้เกิดปัญหา** เพื่อที่เราจะสามารถช่วยเหลือได้@wannaphong: Hello! It is a cause from the preprocessing of tokenizer.

For Multi-Cut, we are using regex to group non-thai word (include space) but longest-matching are grouping just English. https://github.com/PyThaiNLP/pythainlp/blob/dev/pythainlp/tokenize/longest.py#L40@wannaphong: I think if I want longest-matching to grouping consecutive spaces into one token like Multi-Cut, it is may rewrite longest-matching. If you interested, you can edit and submit the pull request!@wannaphong: Fixed #1062

Solution patch:
diff --git a/pythainlp/tokenize/longest.py b/pythainlp/tokenize/longest.py
index 8bd21bf4..1b0bf718 100644
--- a/pythainlp/tokenize/longest.py
+++ b/pythainlp/tokenize/longest.py
@@ -38,6 +38,7 @@ _REAR_DEP_CHAR = ["ั", "ื", "เ", "แ", "โ", "ใ", "ไ", "ํ"]
 _TRAILING_CHAR = ["ๆ", "ฯ"]
 
 _RE_NONTHAI = re.compile(r"[A-Za-z\d]*")
+_RE_SPACES = re.compile(r"\s+")
 
 _KNOWN = True
 _UNKNOWN = False
@@ -134,7 +135,15 @@ class LongestMatchTokenizer:
                     token_statuses.append(_KNOWN)
                 begin_pos += len(match)
 
-        return tokens
+        # Group consecutive spaces into one token
+        grouped_tokens = []
+        for token in tokens:
+            if token.isspace() and grouped_tokens and grouped_tokens[-1].isspace():
+                grouped_tokens[-1] += token
+            else:
+                grouped_tokens.append(token)
+
+        return grouped_tokens
 
     def tokenize(self, text: str) -> List[str]:
         tokens = self.__segment(text)
diff --git a/pythainlp/tokenize/newmm.py b/pythainlp/tokenize/newmm.py
index cd9f238c..c999fb7b 100644
--- a/pythainlp/tokenize/newmm.py
+++ b/pythainlp/tokenize/newmm.py
@@ -182,7 +182,7 @@ def segment(
         # try to break by space first
         space_idx = sample.rfind(" ")
         if space_idx >= 0:
-            cut_pos = space_idx + 1
+            cut_pos = space_idx + 1 + _TEXT_SCAN_BEGIN
         else:
             tokens = list(_onecut(sample, custom_dict))
             token_max_idx = 0


Test patch:
diff --git a/tests/core/test_tokenize.py b/tests/core/test_tokenize.py
index 054a2ab7..c1fd7ba0 100644
--- a/tests/core/test_tokenize.py
+++ b/tests/core/test_tokenize.py
@@ -390,6 +390,18 @@ class TokenizeTestCase(unittest.TestCase):
             longest_tokenizer.word_tokenize("เฉียบพลัน"),
             ["เฉียบพลัน"],
         )
+        self.assertEqual(
+            longest.segment("ทดสอบ  ทดสอบ  ทดสอบ"),
+            ["ทดสอบ", "  ", "ทดสอบ", "  ", "ทดสอบ"],
+        )
+        self.assertEqual(
+            longest.segment("ทดสอบ  ทดสอบ"),
+            ["ทดสอบ", "  ", "ทดสอบ"],
+        )
+        self.assertEqual(
+            longest.segment("ทดสอบ    ทดสอบ"),
+            ["ทดสอบ", "    ", "ทดสอบ"],
+        )
 
     def test_mm(self):
         self.assertEqual(multi_cut.segment(None), [])