Repository: adbar/simplemma

Issue:
in_target_language can count words twice and return ratios above 1.0
I noticed that in current `main` version, the `in_target_language` function can return ratios above 1.0 when given more than one language and the words in the input are matching several languages.

Example:

    >>> in_target_language('It was a true gift', lang='en')
    1.0
    >>> in_target_language('It was a true gift', lang='de')
    0.6666666666666666
    >>> in_target_language('It was a true gift', lang=('en','de'))
    1.6666666666666665

Simplemma 0.9.1 doesn't have this problem:

    >>> in_target_language('It was a true gift', lang='en')
    1.0
    >>> in_target_language('It was a true gift', lang='de')
    0.6666666666666666
    >>> in_target_language('It was a true gift', lang=('en','de'))
    1.0

It's not just a question of capping the scores at 1.0. The problem is that a single word can count more than once if it happens to match multiple languages. Below is an example that demonstrates the problem. I added nonsense words that don't match any language, but their presence is compensated by words that are counted twice.

Current `main` version:

    >>> in_target_language('It was a true gift xxx yyy', lang=('en','de'))
    1.0

Simplemma 0.9.1:

    >>> in_target_language('It was a true gift xxx yyy', lang=('en','de'))
    0.6

Discussion:
codecov[bot]: ## [Codecov](https://app.codecov.io/gh/adbar/simplemma/pull/114?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Adrien+Barbaresi) Report
> Merging [#114](https://app.codecov.io/gh/adbar/simplemma/pull/114?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Adrien+Barbaresi) (d9514a4) into [main](https://app.codecov.io/gh/adbar/simplemma/commit/fa1d96469ca601b5249b8d5cbb42c1474cfd83bb?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Adrien+Barbaresi) (fa1d964) will **increase** coverage by `0.05%`.
> The diff coverage is `100.00%`.

```diff
@@            Coverage Diff             @@
##             main     #114      +/-   ##
==========================================
+ Coverage   96.62%   96.67%   +0.05%     
==========================================
  Files          33       33              
  Lines         651      661      +10     
==========================================
+ Hits          629      639      +10     
  Misses         22       22              
```


| [Files Changed](https://app.codecov.io/gh/adbar/simplemma/pull/114?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Adrien+Barbaresi) | Coverage Δ | |
|---|---|---|
| [simplemma/language\_detector.py](https://app.codecov.io/gh/adbar/simplemma/pull/114?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Adrien+Barbaresi#diff-c2ltcGxlbW1hL2xhbmd1YWdlX2RldGVjdG9yLnB5) | `97.22% <100.00%> (+0.44%)` | :arrow_up: |


adbar: Thanks!

Patch:
diff --git a/simplemma/language_detector.py b/simplemma/language_detector.py
index 52369a6..f6fab14 100644
--- a/simplemma/language_detector.py
+++ b/simplemma/language_detector.py
@@ -198,14 +198,18 @@ class LanguageDetector:
         Returns:
             float: The proportion of text in the target language(s).
         """
-        return sum(
-            percentage
-            for (
-                lang_code,
-                percentage,
-            ) in self.proportion_in_each_language(text).items()
-            if lang_code != "unk"
-        )
+        tokens = self._token_sampler.sample_text(text)
+        if len(tokens) == 0:
+            return 0
+
+        in_target = 0
+        for token in tokens:
+            for lang_code in self._lang:
+                candidate = self._lemmatization_strategy.get_lemma(token, lang_code)
+                if candidate is not None:
+                    in_target += 1
+                    break
+        return in_target / len(tokens)
 
     def main_language(
         self,


Test patch:
diff --git a/tests/test_language_detector.py b/tests/test_language_detector.py
index 4affdd1..ad68f0c 100644
--- a/tests/test_language_detector.py
+++ b/tests/test_language_detector.py
@@ -108,6 +108,15 @@ def test_in_target_language() -> None:
         == 1.0
     )
 
+    langs = ("en", "de")
+    text = "It was a true gift"
+    assert (
+        LanguageDetector(lang=langs).proportion_in_target_languages(text)
+        == in_target_language(text, lang=langs)
+        == 1.0
+    )
+    in_target_language("It was a true gift", lang=("en", "de"))
+
 
 def test_main_language():
     text = "Dieser Satz ist auf Deutsch."