Repository: UAlbertaALTLab/morphodict (language: python)

Issue:
English search takes forever again and returns way too many results
Try searching for "it happened" (I won't provide a handy link because I fear some crawler will visit it and unintentionally take down the site).

It about 7 seconds for the request to be ready, and additional time to actually download the page. Compare this to, say, "[love](https://itwewina.altlab.app/search?q=love)", take takes less than a second for the response headers to be sent and the body to download on my home internet connection. Luckily, after the wait, "ispayin" (what I expected to be the best result) is the first result. So at least the search quality is good!

My hypothesis: the search results are just gathering ALL target language definitions (i.e, English) that contain "it" and adding it the result. Ctrl+F in Chrome tells me that "it" can be found 6052 times on the page 🙃 


Discussion / hints:
My original proposal for the logic with phrase translation was that, 1) if multiple words could be analyzed as either a noun or verb phrase; 2) then the function words should get analyzed as English features (here: _it_) that are converted into Cree ones in the autotranslation (here: `PV/ki+`); and 3) the actual search should be undertaken with only the remaining lexical elements (here: _happened_ -> _happens_). A further consideration, not yet implemented, would be to restrict the results of the (lexical) search to verbs or nouns based on the phrase analysis, if not actually even more narrowly to the individual transitivity/animacy classes that the phrase analysis provides (II, AI, TI, TA; NB. this would not really work for nouns).

For _it happened_, the English phrase analyzer works properly, cf. https://itwewina.altlab.app/admin/fst-tool?text=it+happened&Analyze=Submit+Query - but the search in itwêwina still leaves 'it' among the search words - which probably, and apparently is not good (cf. the verbose results).
As a further comment for _it_, similar to being interpreted as a function word / feature (and not used in the actual search as a lexical element) when occurring as an existential subject, when used as the object for a verb, this should result in the phrase being analyzed as VTI (in contrast to AI and TA) with an inanimate object (`+0SgO`, which can be ignored in the Cree word form generation), cf. https://itwewina.altlab.app/admin/fst-tool?text=she+eats+it&Analyze=Submit+Query, where the pronoun should neither be part of the search. However, for 'she eats it', _it_ still appears to hang around in the search, slowing it down and not yielding the intended result.
And finally, now having it as an object results in the following error/warning:  `"espt_analysis_error": "UnknownTagError('+0SgO')"` (which didn't appear to be the case just a while ago).

We can drop `+0SgO` fairly easily, but I think there are a few issues with the `+0Sg` case.

  - The phrase analyzer FST analyzes `it happened` as `+V+II+0Sg` but the dictionary FST doesn’t generate anything for `ispayin+V+II+Ind+0Sg`. Should `+0Sg` in an English phrase map to `+3Sg` in a Cree wordform?

  - The phrase analyzer FST turns `it was night` into `is night +V+II+Prt+0Sg`. In addition to the above concerns about `+0Sg`, the phrase analyzer should probably drop the `is` word too.

Fix patch:
diff --git a/src/CreeDictionary/API/search/espt_crk.py b/src/CreeDictionary/API/search/espt_crk.py
index dd876592a..1a48df38f 100644
--- a/src/CreeDictionary/API/search/espt_crk.py
+++ b/src/CreeDictionary/API/search/espt_crk.py
@@ -92,9 +92,6 @@ def passthrough_tags_to_tuples(passthrough_tags):
 
 
 verb_tag_map = TagMap(
-    # Person
-    ("+0Sg", "+3Sg", 2),
-    ("+0SgO", "", 3),
     # Tense
     ("+Prt", ("PV/ki+", "+Ind"), 1),  # Preterite aka simple past
     ("+Cond", ("+Fut", "+Cond"), 1),  # Future conditional
@@ -106,6 +103,10 @@ def passthrough_tags_to_tuples(passthrough_tags):
     ("+Def", ("PV/ka+", "+Ind"), 1),
     ("+Inf", ("PV/ka+", "+Cnj"), 1),
     (TagMap.DEFAULT, "+Ind", 1),
+    # Person - see https://github.com/UAlbertaALTLab/cree-intelligent-dictionary/issues/891
+    ("+0Sg", "+3Sg", 2),
+    # Person - object
+    ("+0SgO", (), 3),
     # TODO: also handle "+Inf": ("PV/ta+", "+Cnj")  # future definite?
     *passthrough_tags_to_tuples(verb_passthrough_tags)
 )
@@ -113,18 +114,3 @@ def passthrough_tags_to_tuples(passthrough_tags):
 noun_tag_map = TagMap(
     ("+Dim", "+Der/Dim", 2), *passthrough_tags_to_tuples(noun_passthrough_tags)
 )
-
-
-crkeng_tag_dict = {
-    "+Prt": ("PV/ki+", "+Ind"),  # Preterite aka simple past
-    "+Cond": ("+Fut", "+Cond"),  # Future conditional
-    "+Imm": ("+Imp", "+Imm"),  # Immediate imperative
-    "+Del": ("+Imp", "+Del"),  # Delayed imperative
-    "+Fut": ("PV/wi+", "+Ind"),  # Future
-    # TODO: also handle "+Fut": "PV/wi+",  # Also accept PV/wi without independent as future?
-    # Note that these crk features as disjoint, but both are needed for the eng feature
-    "+Def": ("PV/ka+", "+Ind"),
-    "+Inf": ("PV/ka+", "+Cnj"),
-    # TODO: also handle "+Inf": ("PV/ta+", "+Cnj")  # future definite?
-    "+Dim": ("+Der/Dim",),
-}


Test patch:
diff --git a/src/CreeDictionary/API/search/test_espt_crk.py b/src/CreeDictionary/API/search/test_espt_crk.py
index 3cbde54cd..d9279b4f5 100644
--- a/src/CreeDictionary/API/search/test_espt_crk.py
+++ b/src/CreeDictionary/API/search/test_espt_crk.py
@@ -30,7 +30,7 @@ def test_search_with_tags(query, has_tags, tags, filtered_query):
             {
                 "expected_query_terms": ["crawls"],
                 "expected_new_tags": ["+V", "+AI", "PV/ki+", "+Ind", "+3Pl"],
-                "lemma": "pimitâcimow",
+                "slug": "pimitâcimow",
                 "expected_inflection": "kî-pimitâcimowak",
             },
         ],
@@ -39,7 +39,7 @@ def test_search_with_tags(query, has_tags, tags, filtered_query):
             {
                 "expected_query_terms": ["see"],
                 "expected_new_tags": ["+V", "+TA", "PV/ki+", "+Ind", "+3Pl", "+1SgO"],
-                "lemma": "wâpamêw",
+                "slug": "wâpamêw",
                 "expected_inflection": "nikî-wâpamikwak",
             },
         ],
@@ -48,7 +48,7 @@ def test_search_with_tags(query, has_tags, tags, filtered_query):
             {
                 "expected_query_terms": ["bear"],
                 "expected_new_tags": ["+N", "+Der/Dim", "+Px1Sg", "+Pl"],
-                "lemma": "maskwa",
+                "slug": "maskwa",
                 "expected_inflection": "nimaskomisak",
             },
         ],
@@ -57,17 +57,35 @@ def test_search_with_tags(query, has_tags, tags, filtered_query):
             {
                 "expected_query_terms": ["see"],
                 "expected_new_tags": ["+V", "+AI", "+Ind", "+3Pl"],
-                "lemma": "wâpiw",
+                "slug": "wâpiw",
                 "expected_inflection": "wâpiwak",
             },
         ],
+        [
+            "you ate it",
+            {
+                "expected_query_terms": ["eat"],
+                "expected_new_tags": ["+V", "+TI", "PV/ki+", "+Ind", "+2Sg"],
+                "slug": "mîciw",
+                "expected_inflection": "kikî-mîcin",
+            },
+        ],
+        [
+            "it will fall short",
+            {
+                "expected_query_terms": ["fall", "short"],
+                "expected_new_tags": ["+V", "+II", "PV/ka+", "+Ind", "+3Sg"],
+                "slug": "nôhtêpayiw@vii",
+                "expected_inflection": "ka-nôhtêpayiw",
+            },
+        ],
         [
             "bear",
             {
                 # Don’t try to inflect results for searches not analyzable as phrases
                 "expected_query_terms": ["bear"],
                 "expected_new_tags": [],
-                "lemma": "maskwa",
+                "slug": "maskwa",
                 "expected_inflection": "maskwa",
             },
         ],
@@ -81,7 +99,7 @@ def test_espt_search(db, search, params):
     assert search_run.query.query_string == " ".join(params["expected_query_terms"])
     assert espt_search.new_tags == params["expected_new_tags"]
 
-    lemma1 = Wordform.objects.get(text=params["lemma"], is_lemma=True)
+    lemma1 = Wordform.objects.get(slug=params["slug"], is_lemma=True)
 
     search_run.add_result(
         Result(