Repository: translate/pootle

Issue:
Local TM results are a large edit distance from the source text
Current Local TM is giving matches based on Elasticsearches Levenshtein distance.  The matches do have similarity but in quite a few cases the distance is so great that its not a useful suggestion.  There are cases where substrings within the match prove useful (and indicate the likely reason the match was found).

It does seems that ES's idea of Levenshtein is different from what we are used to.  So we either need delve deaper to find some setting to alter or apply our own Levenshtein to the matches proposed.



Discussion:
From my understanding, since ad7439c there's a sort of _regression_ in this aspect: result sets were previously sliced [based on their relevance score](https://www.elastic.co/guide/en/elasticsearch/guide/current/relevance-intro.html#relevance-intro), and therefore users see matches that can be close as well as really far from the source text.

Notice [how relevance score was being considered](https://github.com/translate/pootle/commit/173f201) in the past. Also note that [minimum score filtering can also be done when querying](https://www.elastic.co/guide/en/elasticsearch/reference/1.4/search-request-min-score.html). Nevertheless I'm not quite sure whether we should [rely on score when doing fuzzy matching or not](https://www.elastic.co/guide/en/elasticsearch/guide/current/fuzzy-scoring.html).

Also I admit that these might smell like _hacky_ ways to slice the obtained results, instead of making use of the power of the search syntax to have more accurate results.  
For instance if I add `"operator": "and"` in the `"match": "source"` section, it feels like I can remove the results that are very far from the source text; some other times texts that could be close enough to the source text go missing though.

So while I don't have a definite answer on how to proceed here, what is clear to me is that `MIN_SCORE`, with its initial semantics, is not meant to be used as a parameter to `fuzziness`, and it's misleading to say the least.

Added to this last point, note that using `fuzziness` values in the `[0.0 .. 1.0]` range [are not supported since version 2.0](https://github.com/elastic/elasticsearch/issues/10638), and here Pootle should probably always use [`AUTO`, as suggested by the docs](https://www.elastic.co/guide/en/elasticsearch/reference/2.0/common-options.html#_string_fields).

In case someone wants to experiment and further debug this, [I committed a change](https://github.com/translate/pootle/commit/73d3fe31d1e9d69981d49112ac67e011d9e2e441) that allows seeing the score of the TM matches as returned by ES in the DOM structure of the results.

Another thought to improve the quality of results would be to also index the length of source texts, and then limiting to a threshold length when querying.

For more background information on the topic, here's an article that explains [how fuzzy searches work on ES](https://www.elastic.co/blog/found-fuzzy-search). Also here's [how Translatewiki makes use of ES](https://phabricator.wikimedia.org/diffusion/ETRA/browse/master/ttmserver/ElasticSearchTTMServer.php) in their TM feature.

> Notice [how relevance score was being considered](https://github.com/translate/pootle/commit/173f201) in the past. Also note that [minimum score filtering can also be done when querying](https://www.elastic.co/guide/en/elasticsearch/reference/1.4/search-request-min-score.html). Nevertheless I'm not quite sure whether we should [rely on score when doing fuzzy matching or not](https://www.elastic.co/guide/en/elasticsearch/guide/current/fuzzy-scoring.html).

Minimum score filtering cannot be done when querying because [we are altering the results score by the weight of their TM](https://github.com/translate/pootle/blob/master/pootle/core/search/backends/elasticsearch.py#L113).

So as mentioned in some chat, the Levenshtein distance is used when matching **terms** individually, not taking into account the entire text as a whole. At the same time the fuzziness doesn't account for result's scoring purposes.

As [I already linked before](https://phabricator.wikimedia.org/diffusion/ETRA/browse/master/ttmserver/ElasticSearchTTMServer.php), Wikipedia's Translatewiki overcomes this by [boosting the score](https://phabricator.wikimedia.org/diffusion/ETRA/browse/master/ttmserver/ElasticSearchTTMServer.php;ee54b9ac3369918054b1cdea20683c41c471ec4f$64-72) with the Levenshtein distance applied to the full text. The boosting is done via a function which is implemented in a custom ES plugin.

Since this is overkill in our use case, after talking with @iafan we think we can filter out the matches proposed by ES which are really far away from the source text. As an initial heuristic let's say that we will accept results that change a third part of the source text at max, and discard the rest because we consider they are irrelevant. This should be enough to overcome the core issue we are facing here.

> Since this is overkill in our use case, after talking with @iafan we think we can filter out the matches proposed by ES which are really far away from the source text. As an initial heuristic let's say that we will accept results that change a third part of the source text at max, and discard the rest because we consider they are irrelevant. This should be enough to overcome the core issue we are facing here.

A third part looks ok to me. I assume this would be done after retrieving all the results and altering their score with the TM weight.

> I assume this would be done after retrieving all the results and altering their score with the TM weight.

No score will be altered — as said, custom plugins are needed for doing so using the Levenshtein distance. We will post-process the results to filter out those items that are too far from the original string.


Resolving patch:
diff --git a/pootle/checks.py b/pootle/checks.py
--- a/pootle/checks.py
+++ b/pootle/checks.py
@@ -321,14 +321,6 @@ def check_settings(app_configs=None, **kwargs):
                     id="pootle.C012",
                 ))
 
-            if 'MIN_SCORE' not in settings.POOTLE_TM_SERVER[server]:
-                errors.append(checks.Critical(
-                    _("POOTLE_TM_SERVER['%s'] has no MIN_SCORE.", server),
-                    hint=_("Set a MIN_SCORE for POOTLE_TM_SERVER['%s'].",
-                           server),
-                    id="pootle.C013",
-                ))
-
             if ('WEIGHT' in settings.POOTLE_TM_SERVER[server] and
                 not (0.0 <= settings.POOTLE_TM_SERVER[server]['WEIGHT']
                      <= 1.0)):
diff --git a/pootle/core/search/backends/elasticsearch.py b/pootle/core/search/backends/elasticsearch.py
--- a/pootle/core/search/backends/elasticsearch.py
+++ b/pootle/core/search/backends/elasticsearch.py
@@ -11,6 +11,8 @@
 
 import logging
 
+import Levenshtein
+
 try:
     from elasticsearch import Elasticsearch
     from elasticsearch.exceptions import ElasticsearchException
@@ -26,6 +28,42 @@
 logger = logging.getLogger(__name__)
 
 
+DEFAULT_MIN_SIMILARITY = 0.7
+
+
+def filter_hits_by_distance(hits, source_text,
+                            min_similarity=DEFAULT_MIN_SIMILARITY):
+    """Returns ES `hits` filtered according to their Levenshtein distance
+    to the `source_text`.
+
+    Any hits with a similarity value (0..1) lower than `min_similarity` will be
+    discarded. It's assumed that `hits` is already sorted from higher to lower
+    score.
+    """
+    if min_similarity <= 0 or min_similarity >= 1:
+        min_similarity = DEFAULT_MIN_SIMILARITY
+
+    filtered_hits = []
+    for hit in hits:
+        hit_source_text = hit['_source']['source']
+        distance = Levenshtein.distance(source_text, hit_source_text)
+        similarity = (
+            1 - distance / float(max(len(source_text), len(hit_source_text)))
+        )
+
+        logger.debug(
+            'Similarity: %.2f (distance: %d)\nOriginal:\t%s\nComparing with:\t%s',
+            similarity, distance, source_text, hit_source_text
+        )
+
+        if similarity < min_similarity:
+            break
+
+        filtered_hits.append(hit)
+
+    return filtered_hits
+
+
 class ElasticSearchBackend(SearchBackend):
     def __init__(self, config_name):
         super(ElasticSearchBackend, self).__init__(config_name)
@@ -74,7 +112,7 @@ def search(self, unit):
                     "match": {
                         "source": {
                             "query": unit.source,
-                            "fuzziness": self._settings['MIN_SCORE'],
+                            "fuzziness": 'AUTO',
                         }
                     }
                 }
@@ -92,7 +130,13 @@ def search(self, unit):
                          self._settings["PORT"], unit)
             return []
 
-        for hit in es_res['hits']['hits']:
+        hits = filter_hits_by_distance(
+            es_res['hits']['hits'],
+            unit.source,
+            min_similarity=self._settings.get('MIN_SIMILARITY',
+                                              DEFAULT_MIN_SIMILARITY)
+        )
+        for hit in hits:
             if self._is_valuable_hit(unit, hit):
                 body = hit['_source']
                 translation_pair = body['source'] + body['target']


Test patch: