Repository: DoodleBears/split-lang (language: python)

Issue:
Revise possible_detection_list using `fast_langdetect.detect_multilingual`, improve accuracy in Chinese and Japanese
# Summary

- `possible_detection_list` will return possible detected languages
- This was used when near substrings are in same language `A` and the middle substring been detect as other language `B` WHILE its possible language including the language `A`
	- e.g. 日语使用者应超过一亿三千万人 -> 日语使用者应超过一亿 | 三千万 | 人
		- zh: 日语使用者应超过一亿
		- ja: 三千万人 （could be ja or zh）
Revise possible_detection_list using `fast_langdetect.detect_multilingual`, improve accuracy in Chinese and Japanese
# Summary

- `possible_detection_list` will return possible detected languages
- This was used when near substrings are in same language `A` and the middle substring been detect as other language `B` WHILE its possible language including the language `A`
	- e.g. 日语使用者应超过一亿三千万人 -> 日语使用者应超过一亿 | 三千万 | 人
		- zh: 日语使用者应超过一亿
		- ja: 三千万人 （could be ja or zh）


Discussion / hints:

Fix patch:
diff --git a/setup.py b/setup.py
index e5ad51b..b2372b4 100644
--- a/setup.py
+++ b/setup.py
@@ -24,10 +24,10 @@ def read(*relpath):
     packages=find_packages(),
     install_requires=[
         "fast_langdetect",
-        "lingua-language-detector",
         "pydantic",
         "budoux",
         "wordfreq",
+        "wordfreq[cjk]",
     ],
     classifiers=[
         "Programming Language :: Python :: 3",
diff --git a/split-lang-benchmark.ipynb b/split-lang-benchmark.ipynb
index 3f4b02f..4baa769 100644
--- a/split-lang-benchmark.ipynb
+++ b/split-lang-benchmark.ipynb
@@ -9,7 +9,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 11,
+   "execution_count": 80,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -27,21 +27,9 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 12,
+   "execution_count": 81,
    "metadata": {},
-   "outputs": [
-    {
-     "name": "stderr",
-     "output_type": "stream",
-     "text": [
-      "c:\\Users\\admin\\.conda\\envs\\melotts\\lib\\site-packages\\wtpsplit\\__init__.py:45: DeprecationWarning: You are using WtP, the old sentence segmentation model. It is highly encouraged to use SaT instead due to strongly improved performance and efficiency. See https://github.com/segment-any-text/wtpsplit for more info. To ignore this warning, set ignore_legacy_warning=True.\n",
-      "  warnings.warn(\n",
-      "c:\\Users\\admin\\.conda\\envs\\melotts\\lib\\site-packages\\sklearn\\base.py:376: InconsistentVersionWarning: Trying to unpickle estimator LogisticRegression from version 1.2.2 when using version 1.5.0. This might lead to breaking code or invalid results. Use at your own risk. For more info please refer to:\n",
-      "https://scikit-learn.org/stable/model_persistence.html#security-maintainability-limitations\n",
-      "  warnings.warn(\n"
-     ]
-    }
-   ],
+   "outputs": [],
    "source": [
     "from wtpsplit import SaT, WtP\n",
     "sat = SaT(\"sat-1l-sm\")\n",
@@ -52,7 +40,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 13,
+   "execution_count": 82,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -61,14 +49,14 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 14,
+   "execution_count": 83,
    "metadata": {},
    "outputs": [
     {
      "name": "stderr",
      "output_type": "stream",
      "text": [
-      "100%|██████████| 1/1 [00:00<00:00, 166.68it/s]\n"
+      "100%|██████████| 1/1 [00:00<00:00, 124.43it/s]\n"
      ]
     },
     {
@@ -77,7 +65,7 @@
        "['你', '喜欢看', 'アニメ', '吗']"
       ]
      },
-     "execution_count": 14,
+     "execution_count": 83,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -88,7 +76,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 15,
+   "execution_count": 84,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -183,7 +171,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 16,
+   "execution_count": 85,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -225,7 +213,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 17,
+   "execution_count": 86,
    "metadata": {},
    "outputs": [
     {
@@ -372,7 +360,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 18,
+   "execution_count": 87,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -386,7 +374,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 19,
+   "execution_count": 88,
    "metadata": {},
    "outputs": [
     {
@@ -460,45 +448,94 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 20,
+   "execution_count": 116,
    "metadata": {},
    "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "c:\\Users\\admin\\.conda\\envs\\melotts\\lib\\site-packages\\wtpsplit\\__init__.py:45: DeprecationWarning: You are using WtP, the old sentence segmentation model. It is highly encouraged to use SaT instead due to strongly improved performance and efficiency. See https://github.com/segment-any-text/wtpsplit for more info. To ignore this warning, set ignore_legacy_warning=True.\n",
+      "  warnings.warn(\n"
+     ]
+    },
     {
      "name": "stdout",
      "output_type": "stream",
      "text": [
-      "['Vielen ', 'Dank ', 'merci ', 'beaucoup ', 'for your help.']\n",
-      "de:Vielen |nl:Dank |fr:merci |fr:beaucoup |en:for your help.|\n",
-      "0.5008478164672852\n",
-      "de:Vielen|nl:Dank|fr:merci|fr:beaucoup|en:for|en:your|en:help.|\n",
+      "['Ich ', 'bin müde ', 'je suis fatigué ', 'and ', 'I ', 'need some rest']\n",
+      "\n",
+      "0.8507270812988281\n",
+      "['Ich', 'bin', 'müde', 'je', 'suis', 'fatigué', 'and', 'I', 'need', 'some', 'rest']\n",
+      "Ich: ['de:1.0']\n",
+      "\n",
+      "bin: ['en:0.394521027803421', 'id:0.21292246878147125', 'tr:0.15927168726921082', 'ms:0.09677533060312271', 'eo:0.030221346765756607', 'jv:0.023466553539037704', 'sq:0.013604077510535717', 'sv:0.012493844144046307']\n",
+      "\n",
+      "müde: ['de:0.9626638293266296', 'tr:0.026752416044473648']\n",
+      "\n",
+      "je: ['sr:0.8350609540939331', 'fr:0.15938909351825714']\n",
+      "\n",
+      "suis: ['fr:0.9970543384552002']\n",
+      "\n",
+      "fatigué: ['fr:0.7745229601860046', 'es:0.14570455253124237', 'nl:0.03426196426153183', 'de:0.014598727226257324']\n",
+      "\n",
+      "and: ['en:0.9983668923377991']\n",
+      "\n",
+      "I: ['en:0.9979130625724792']\n",
+      "\n",
+      "need: ['en:0.9885499477386475']\n",
+      "\n",
+      "some: ['en:0.993599534034729']\n",
+      "\n",
+      "rest: ['en:0.46527764201164246', 'nl:0.05554379150271416', 'ja:0.039999693632125854', 'pl:0.033725958317518234', 'ar:0.028888104483485222', 'sv:0.028634123504161835', 'no:0.024481678381562233', 'id:0.022417806088924408', 'tr:0.02039233408868313', 'zh:0.014322302304208279']\n",
+      "\n",
+      "\n",
       "0.0\n"
      ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "c:\\Users\\admin\\.conda\\envs\\melotts\\lib\\site-packages\\sklearn\\base.py:376: InconsistentVersionWarning: Trying to unpickle estimator LogisticRegression from version 1.2.2 when using version 1.5.0. This might lead to breaking code or invalid results. Use at your own risk. For more info please refer to:\n",
+      "https://scikit-learn.org/stable/model_persistence.html#security-maintainability-limitations\n",
+      "  warnings.warn(\n"
+     ]
     }
    ],
    "source": [
     "from wtpsplit import WtP\n",
     "from time import time\n",
     "from datetime import datetime\n",
-    "text = \"Vielen Dank merci beaucoup for your help.\"\n",
-    "\n",
+    "# text = \"Vielen Dank merci beaucoup for your help.\"\n",
+    "text = \"Ich bin müde je suis fatigué and I need some rest\"\n",
+    "# text = \"日语使用者应超过一亿三千万人\"\n",
+    "# text = \"我是 VGroupChatBot，一个旨在支持多人通信的助手，通过可视化消息来帮助团队成员更好地交流。我可以帮助团队成员更好地整理和共享信息，特别是在讨论、会议和Brainstorming等情况下。你好我的名字是西野くまですmy name is bob很高兴认识你どうぞよろしくお願いいたします「こんにちは」是什么意思。我的名字是西野くまです。I am from Tokyo, 日本の首都。今天的天气非常好\"\n",
     "time1 = datetime.now().timestamp()\n",
     "wtp = WtP('wtp-bert-mini')\n",
-    "substrings = wtp.split(text_or_texts=text, threshold=1e-4)\n",
+    "substrings = wtp.split(text_or_texts=text, threshold=2e-3)\n",
     "print(substrings)\n",
     "for substring in substrings:\n",
     "    # lang = lingua_lang_detect_all(substring)\n",
-    "    lang = fast_lang_detect(substring)\n",
-    "    print(f\"{lang}:{substring}\",end='|')\n",
+    "    lang = lingua_lang_detect_all(substring)\n",
+    "    # print(f\"{lang}:{substring}\",end='|')\n",
     "print()\n",
     "time2 = datetime.now().timestamp()\n",
     "\n",
     "print(time2 - time1)\n",
     "\n",
+    "from split_lang import LangSplitter\n",
+    "lang_splitter = LangSplitter()\n",
+    "substrings = lang_splitter._parse_without_zh_ja(text=text)\n",
     "substrings = text.split(' ')\n",
+    "# substrings = lang_splitter._parse_zh_ja(text=text)\n",
+    "# substrings = lang_splitter._parse_without_zh_ja(text=text)\n",
+    "print(substrings)\n",
     "for substring in substrings:\n",
     "    # lang = lingua_lang_detect_all(substring)\n",
-    "    lang = fast_lang_detect(substring)\n",
-    "    print(f\"{lang}:{substring}\",end='|')\n",
+    "    lang = fast_langdetect.detect_multilingual(substring, low_memory=False, k=10, threshold=0.01)\n",
+    "    lang_list = [f\"{item['lang']}:{item['score']}\" for item in lang]\n",
+    "    print(f\"{substring}: {lang_list}\",end='\\n\\n')\n",
     "time3 = datetime.now().timestamp()\n",
     "print()\n",
     "print(time3 - time2)"
@@ -506,23 +543,23 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 29,
+   "execution_count": 117,
    "metadata": {},
    "outputs": [
     {
      "name": "stdout",
      "output_type": "stream",
      "text": [
-      "9.55e-05\n",
-      "7.82e-05\n",
-      "1.221227621483376\n"
+      "1.66e-05\n",
+      "0.00126\n",
+      "0.013174603174603174\n"
      ]
     }
    ],
    "source": [
     "from wordfreq import word_frequency\n",
-    "ja_freq = word_frequency('日本人', 'ja')\n",
-    "zh_freq = word_frequency('日本人', 'zh')\n",
+    "ja_freq = word_frequency('bin ', 'en')\n",
+    "zh_freq = word_frequency('bin ', 'de')\n",
     "print(ja_freq)\n",
     "print(zh_freq)\n",
     "print(ja_freq / zh_freq)"
diff --git a/split_lang/detect_lang/__init__.py b/split_lang/detect_lang/__init__.py
index 54e1c88..1717460 100644
--- a/split_lang/detect_lang/__init__.py
+++ b/split_lang/detect_lang/__init__.py
@@ -1,5 +1,5 @@
 from .detector import (
     detect_lang_combined,
     possible_detection_list,
-    is_word_freq_higher_in_ja,
+    is_word_freq_higher_in_lang_b,
 )
diff --git a/split_lang/detect_lang/detector.py b/split_lang/detect_lang/detector.py
index 6cb3170..9ed3d40 100644
--- a/split_lang/detect_lang/detector.py
+++ b/split_lang/detect_lang/detector.py
@@ -1,20 +1,11 @@
 import logging
 from typing import List
 import fast_langdetect
-from lingua import LanguageDetectorBuilder
 from wordfreq import word_frequency
 
 from ..model import LangSectionType
 from ..split.utils import contains_ja
 
-
-all_detector = (
-    LanguageDetectorBuilder.from_all_languages()
-    .with_preloaded_language_models()
-    .build()
-)
-
-
 logger = logging.getLogger(__name__)
 
 
@@ -24,13 +15,6 @@ def fast_lang_detect(text: str) -> str:
     return result
 
 
-def lingua_lang_detect_all(text: str) -> str:
-    language = all_detector.detect_language_of(text=text)
-    if language is None:
-        return "x"
-    return language.iso_code_639_1.name.lower()
-
-
 # For example '衬衫' cannot be detected by `langdetect`, and `fast_langdetect` will detect it as 'en'
 def detect_lang_combined(text: str, lang_section_type: LangSectionType) -> str:
     if lang_section_type is LangSectionType.ZH_JA:
@@ -41,9 +25,15 @@ def detect_lang_combined(text: str, lang_section_type: LangSectionType) -> str:
 
 
 def possible_detection_list(text) -> List[str]:
-    languages = []
-    languages.append(fast_lang_detect(text))
-    languages.append(lingua_lang_detect_all(text))
+    languages = [
+        item["lang"]
+        for item in fast_langdetect.detect_multilingual(
+            text,
+            low_memory=False,
+            k=5,
+            threshold=0.01,
+        )
+    ]
     return languages
 
 
@@ -51,10 +41,12 @@ def _detect_word_freq_in_lang(word: str, lang: str) -> float:
     return word_frequency(word=word, lang=lang)
 
 
-def is_word_freq_higher_in_ja(word: str) -> bool:
-    word_freq_ja = _detect_word_freq_in_lang(word=word, lang="ja")
-    word_freq_zh = _detect_word_freq_in_lang(word=word, lang="zh")
-    if word_freq_zh == 0:
+def is_word_freq_higher_in_lang_b(word: str, lang_a: str, lang_b: str) -> bool:
+    if lang_a == "x" or lang_b == "x":
+        return False
+    word_freq_lang_b = _detect_word_freq_in_lang(word=word, lang=lang_b)
+    word_freq_lang_a = _detect_word_freq_in_lang(word=word, lang=lang_a)
+    if word_freq_lang_a == 0:
         return False
     # 0.8 means either is more frequently used in Japanese or in both language the word is frequently used
-    return (word_freq_ja / word_freq_zh) > 0.8
+    return (word_freq_lang_b / word_freq_lang_a) > 0.8
diff --git a/split_lang/split/splitter.py b/split_lang/split/splitter.py
index 91d920c..e1c7d4a 100644
--- a/split_lang/split/splitter.py
+++ b/split_lang/split/splitter.py
@@ -10,7 +10,7 @@
 from ..detect_lang.detector import (
     detect_lang_combined,
     possible_detection_list,
-    is_word_freq_higher_in_ja,
+    is_word_freq_higher_in_lang_b,
 )
 from ..model import LangSectionType, SubString, SubStringSection
 from .utils import PUNCTUATION, contains_hangul, contains_zh_ja, contains_ja
@@ -195,7 +195,7 @@ def _parse_without_zh_ja(self, text: str):
         return words
 
     # MARK: _parse_zh_ja
-    def _parse_zh_ja(self, text):
+    def _parse_zh_ja(self, text) -> List[str]:
         splitted_texts_jp = []
         splitted_texts_jp = jp_budoux_parser.parse(text)
 
@@ -353,8 +353,9 @@ def _is_merge_middle_to_two_side(
         is_middle_zh_side_ja_and_middle_is_high_freq_in_ja = (
             left.lang == "ja"
             and middle.lang == "zh"
-            and is_word_freq_higher_in_ja(middle.text)
+            and is_word_freq_higher_in_lang_b(middle.text, middle.lang, "ja")
         )
+
         return (
             middle_lang_is_possible_the_same_as_left
             or middle_lang_is_x
@@ -367,15 +368,26 @@ def _is_cur_short_and_near_long(self, cur: SubString, near: SubString):
             cur.length <= 2 and near.length >= 6 and near.lang == "zh"
         )
         # e.g. 日本人, 今晚, 国外移民
+        cur_lang_is_possible_the_same_as_near = near.lang in possible_detection_list(
+            cur.text
+        )
         is_cur_short_and_near_is_ja_and_middle_is_high_freq_in_ja = (
-            cur.lang == "zh"
-            and cur.length <= 4
+            cur.length <= 4
+            and cur.lang == "zh"
             and near.lang == "ja"
-            and is_word_freq_higher_in_ja(cur.text)
+            and is_word_freq_higher_in_lang_b(cur.text, cur.lang, "ja")
+        )
+        is_cur_short_and_near_is_zh_and_middle_is_high_freq_in_zh = (
+            cur.length <= 4
+            and cur.lang == "ja"
+            and near.lang == "zh"
+            and is_word_freq_higher_in_lang_b(cur.text, cur.lang, "zh")
         )
         return (
             is_cur_short_and_near_is_zh_and_long
+            or cur_lang_is_possible_the_same_as_near
             or is_cur_short_and_near_is_ja_and_middle_is_high_freq_in_ja
+            or is_cur_short_and_near_is_zh_and_middle_is_high_freq_in_zh
         )
 
     # MARK: _merge_middle_substr_to_two_side
@@ -408,9 +420,10 @@ def _merge_side_substr_to_near(self, substrings: List[SubString]):
                 substrings[0], substrings[1]
             )
 
-            is_possible_same_lang_with_near = substrings[
-                1
-            ].lang in possible_detection_list(substrings[0].text)
+            is_possible_same_lang_with_near = (
+                substrings[1].lang in possible_detection_list(substrings[0].text)
+                and substrings[1].length <= 5
+            )
 
         is_need_merge_to_right = (
             is_lang_x or is_cur_short_and_near_long or is_possible_same_lang_with_near
@@ -428,9 +441,10 @@ def _merge_side_substr_to_near(self, substrings: List[SubString]):
             is_cur_short_and_near_long = self._is_cur_short_and_near_long(
                 substrings[-1], substrings[-2]
             )
-            is_possible_same_lang_with_near = substrings[
-                -2
-            ].lang in possible_detection_list(substrings[-1].text)
+            is_possible_same_lang_with_near = (
+                substrings[-2].lang in possible_detection_list(substrings[-1].text)
+                and substrings[-2].length <= 5
+            )
 
         is_need_merge_to_left = (
             is_lang_x or is_cur_short_and_near_long or is_possible_same_lang_with_near
@@ -631,6 +645,7 @@ def _smart_concat_logic(
         lang_section_type: LangSectionType,
     ):
 
+        lang_text_list = self._merge_substrings(lang_text_list)
         lang_text_list = self._merge_middle_substr_to_two_side(lang_text_list)
         lang_text_list = self._merge_substrings(lang_text_list)
         lang_text_list = self._get_languages(


Test patch:
diff --git a/tests/data/correct_split_merge_punc.txt b/tests/data/correct_split_merge_punc.txt
index cf7b49d..3acfc07 100644
--- a/tests/data/correct_split_merge_punc.txt
+++ b/tests/data/correct_split_merge_punc.txt
@@ -1,3 +1,5 @@
+匿名掲示板
+2024年の日本
 你喜欢看|アニメ|吗？
 我的名字是|田中さんです。
 我的名字是田中|さんです。
@@ -5,6 +7,8 @@
 衬衫的价格是9.15便士。
 衬衫的价格是233亿元。
 衬衫的价格是233亿元人民币。
+我给你送的|手紙|你读了吗？
+我给你送的|手紙，|你读了吗？
 你最近好吗|最近どうですか？
 你最近好吗最近|どうですか？
 你最近好吗、|最近どうですか？
@@ -31,6 +35,8 @@
 你会说几种语言、|何ヶ国語話せますか？|몇 개 언어를 할 수 있어요？
 我昨天看了一本书、|その本はとても面白かったです。|어제 책을 읽었는데, 정말 재미있었어요。
 昨天|見た映画はとても感動的でした。|我朋友是日本人|彼はとても優しいです。
+我的名字是|西野くまです。|I am from Tokyo, |日本の首都。|今天的天气非常好
+我是 |VGroupChatBot，|一个旨在支持多人通信的助手，通过可视化消息来帮助团队成员更好地交流。我可以帮助团队成员更好地整理和共享信息，特别是在讨论、会议和|Brainstorming|等情况下。你好我的名字是|西野くまです|my name is bob|很高兴认识你|どうぞよろしくお願いいたします「こんにちは」|是什么意思。
 Bonjour, |wie geht's dir |today?
 Vielen Dank |merci beaucoup |for your help.
 Ich bin müde |je suis fatigué |and I need some rest.
@@ -38,12 +44,8 @@ Ich mag dieses Buch |ce livre est intéressant |and it has a great story.
 Ich mag dieses Buch, |ce livre est intéressant, |and it has a great story.
 The shirt is 9.15 dollars.
 The shirt is 233 dollars.
-我是 |VGroupChatBot，|一个旨在支持多人通信的助手，通过可视化消息来帮助团队成员更好地交流。我可以帮助团队成员更好地整理和共享信息，特别是在讨论、会议和|Brainstorming|等情况下。你好我的名字是|西野くまです|my name is bob|很高兴认识你|どうぞよろしくお願いいたします「こんにちは」|是什么意思。
-我的名字是|西野くまです。|I am from Tokyo, |日本の首都。|今天的天气非常好
-我给你送的|手紙|你读了吗？
 lang-split
 I have 10, |€
-日本のメディアでは「匿名掲示板」であると紹介されることが多いが、2003年1月7日から全書き込みについて|IP|アドレスの記録・保存を始めており、厳密には匿名掲示板ではなくなっていると|CNET |Japan|は報じている
-日本語（にほんご、にっぽんご）は、日本国内や、かつての日本領だった国、そして国外移民や移住者を含む日本人同士の間で使用されている言語。
-2024年の日本
-匿名掲示板
\ No newline at end of file
+日本のメディアでは「匿名掲示板」であると紹介されることが多いが、2003年1月7日から全書き込みについて|IP|アドレスの記録・保存を始めており、厳密には匿名掲示板ではなくなっていると|CNET Japan|は報じている
+日本語（にほんご、にっぽんご）は、日本国内や、かつての日本領だった国、そして国外移民や移住者を含む日本人同士の間で使用されている言語。日本は法令によって公用語を規定していないが、法令その他の公用文は全て日本語で記述され、各種法令において日本語を用いることが規定され、学校教育においては「国語」の教科として学習を行うなど、事実上日本国内において唯一の公用語となっている。
+日语是日本通用语及事实上的官方语言。没有精确的日语使用人口的统计，如果计算日本人口以及居住在日本以外的日本人、日侨和日裔，日语使用者应超过一亿三千万人。
\ No newline at end of file