Repository: adbar/simplemma

Issue:
in_target_language can return ratios above 1.0 (again)
Bug #112 seems to have resurfaced. Again, `in_target_language('It was a true gift', lang=('en','de'))` returns 1.6666... in simplemma 1.1.1.

Looking at the code for `proportion_in_target_languages`, it was changed back to the [buggy algorithm](https://github.com/adbar/simplemma/commit/6586bcb389571a1b9b9a7b43dd8a238dd9656b9c#diff-3d9b26cb1f0cb05a8aa25538052fdc1629e74a04405d432d39cf5d7849f43fb7) that uses `sum()` in #116 (by @juanjoDiaz) and the unit test guarding against this bug was simply [removed](https://github.com/adbar/simplemma/commit/6586bcb389571a1b9b9a7b43dd8a238dd9656b9c#diff-615c74a5aa69e25288b38f719352db39c98a0e3e83c938ea0498c9882465c9ae) in the same PR :(

Discussion:
codecov[bot]: ## [Codecov](https://app.codecov.io/gh/adbar/simplemma/pull/150?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Adrien+Barbaresi) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Project coverage is 97.24%. Comparing base [(`e1fb6a0`)](https://app.codecov.io/gh/adbar/simplemma/commit/e1fb6a07d2896b153b18cacf5bf40803b23115de?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Adrien+Barbaresi) to head [(`0071258`)](https://app.codecov.io/gh/adbar/simplemma/commit/007125882bec698966e97a5fc773fce4b00b9c79?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Adrien+Barbaresi).

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main     #150      +/-   ##
==========================================
+ Coverage   97.19%   97.24%   +0.04%     
==========================================
  Files          35       35              
  Lines         606      616      +10     
==========================================
+ Hits          589      599      +10     
  Misses         17       17              
```

</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/adbar/simplemma/pull/150?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Adrien+Barbaresi).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Adrien+Barbaresi).

adbar: Thanks!

Patch:
diff --git a/simplemma/language_detector.py b/simplemma/language_detector.py
index 30c35be..451c71a 100644
--- a/simplemma/language_detector.py
+++ b/simplemma/language_detector.py
@@ -189,14 +189,18 @@ class LanguageDetector:
         Returns:
             float: The proportion of text in the target language(s).
         """
-        return sum(
-            percentage
-            for (
-                lang_code,
-                percentage,
-            ) in self.proportion_in_each_language(text).items()
-            if lang_code != "unk"
-        )
+        tokens = self._token_sampler.sample_text(text)
+        if len(tokens) == 0:
+            return 0
+
+        in_target = 0
+        for token in tokens:
+            for lang_code in self._lang:
+                candidate = self._lemmatization_strategy.get_lemma(token, lang_code)
+                if candidate is not None:
+                    in_target += 1
+                    break
+        return in_target / len(tokens)
 
     def main_language(
         self,


Test patch:
diff --git a/tests/test_language_detector.py b/tests/test_language_detector.py
index 0530668..b34dfb4 100644
--- a/tests/test_language_detector.py
+++ b/tests/test_language_detector.py
@@ -114,6 +114,14 @@ def test_in_target_language() -> None:
         == 1.0
     )
 
+    langs = ("en", "de")
+    text = "It was a true gift"
+    assert (
+        LanguageDetector(lang=langs).proportion_in_target_languages(text)
+        == in_target_language(text, lang=langs)
+        == 1.0
+    )
+
 
 def test_main_language():
     text = "Dieser Satz ist auf Deutsch."