Repository: adbar/simplemma

Issue:
in_target_language can return ratios above 1.0 (again)
Bug #112 seems to have resurfaced. Again, `in_target_language('It was a true gift', lang=('en','de'))` returns 1.6666... in simplemma 1.1.1.

Looking at the code for `proportion_in_target_languages`, it was changed back to the [buggy algorithm](https://github.com/adbar/simplemma/commit/6586bcb389571a1b9b9a7b43dd8a238dd9656b9c#diff-3d9b26cb1f0cb05a8aa25538052fdc1629e74a04405d432d39cf5d7849f43fb7) that uses `sum()` in #116 (by @juanjoDiaz) and the unit test guarding against this bug was simply [removed](https://github.com/adbar/simplemma/commit/6586bcb389571a1b9b9a7b43dd8a238dd9656b9c#diff-615c74a5aa69e25288b38f719352db39c98a0e3e83c938ea0498c9882465c9ae) in the same PR :(

Discussion:
@adbar: @osma Thank you for reporting the issue, this one slipped through.

Solution patch:
diff --git a/simplemma/language_detector.py b/simplemma/language_detector.py
index 30c35be..451c71a 100644
--- a/simplemma/language_detector.py
+++ b/simplemma/language_detector.py
@@ -189,14 +189,18 @@ class LanguageDetector:
         Returns:
             float: The proportion of text in the target language(s).
         """
-        return sum(
-            percentage
-            for (
-                lang_code,
-                percentage,
-            ) in self.proportion_in_each_language(text).items()
-            if lang_code != "unk"
-        )
+        tokens = self._token_sampler.sample_text(text)
+        if len(tokens) == 0:
+            return 0
+
+        in_target = 0
+        for token in tokens:
+            for lang_code in self._lang:
+                candidate = self._lemmatization_strategy.get_lemma(token, lang_code)
+                if candidate is not None:
+                    in_target += 1
+                    break
+        return in_target / len(tokens)
 
     def main_language(
         self,


Test patch:
diff --git a/tests/test_language_detector.py b/tests/test_language_detector.py
index 0530668..b34dfb4 100644
--- a/tests/test_language_detector.py
+++ b/tests/test_language_detector.py
@@ -114,6 +114,14 @@ def test_in_target_language() -> None:
         == 1.0
     )
 
+    langs = ("en", "de")
+    text = "It was a true gift"
+    assert (
+        LanguageDetector(lang=langs).proportion_in_target_languages(text)
+        == in_target_language(text, lang=langs)
+        == 1.0
+    )
+
 
 def test_main_language():
     text = "Dieser Satz ist auf Deutsch."