Repository: NatLibFi/Annif (language: python)

Issue:
STWFSA backend returns duplicate results
I set up a stwfsa based project using this configuration adapted from the wiki page of the STWFSA backend:

```
[stw-stwfsa-en]
name=STWFSA STW english
language=en
backend=stwfsa
vocab=stw-en
concept_type_uri=http://zbw.eu/namespaces/zbw-extensions/Descriptor
sub_thesaurus_type_uri=http://zbw.eu/namespaces/zbw-extensions/Thsys
thesaurus_relation_type_uri=http://www.w3.org/2004/02/skos/core#broader
thesaurus_relation_is_specialisation=False
```

Then I trained this with stw-econbiz-small.tsv.gz from the Annif-tutorial stw-zbw data set.

When I use `annif suggest` with text that contains multiple matches for the same term, I get duplicate results:

```
$ echo "economics. economics" | annif suggest stw-stwfsa-en
<http://zbw.eu/stw/descriptor/10032-2>	Economics	0.14664009111617313
<http://zbw.eu/stw/descriptor/10032-2>	Economics	0.14664009111617313
```

This is obviously wrong, since the list of suggestions should not contain duplicates.

Looking at [this PR](https://github.com/zbw/stwfsapy/pull/32) that was quite recently merged to stwfsapy, I think this may already have been fixed upstream. Annif would just need to switch to a newer version of stwfsapy (and such a version must be released first). Ping @mo-fu - do you agree with the analysis? Could this be fixed on the Annif side as well by upgrading stwfsapy?


Discussion / hints:
Yes, this was recently fixed upstream. I try to get out a release this week. There are additional changes for handling of longer inputs that I want to include.
Great to hear, thanks @mo-fu!

Fix patch:
diff --git a/annif/backend/stwfsa.py b/annif/backend/stwfsa.py
index 65d57ab08..1ddf8e590 100644
--- a/annif/backend/stwfsa.py
+++ b/annif/backend/stwfsa.py
@@ -20,6 +20,7 @@
     'expand_abbreviation_with_punctuation')
 _KEY_SIMPLE_ENGLISH_PLURAL_RULES = 'simple_english_plural_rules'
 _KEY_INPUT_LIMIT = 'input_limit'
+_KEY_USE_TXT_VEC = 'use_txt_vec'
 
 
 class StwfsaBackend(backend.AnnifBackend):
@@ -40,6 +41,7 @@ class StwfsaBackend(backend.AnnifBackend):
         _KEY_EXPAND_ABBREVIATION_WITH_PUNCTUATION: boolean,
         _KEY_SIMPLE_ENGLISH_PLURAL_RULES: boolean,
         _KEY_INPUT_LIMIT: int,
+        _KEY_USE_TXT_VEC: bool,
     }
 
     DEFAULT_PARAMETERS = {
@@ -57,6 +59,7 @@ class StwfsaBackend(backend.AnnifBackend):
         _KEY_EXPAND_ABBREVIATION_WITH_PUNCTUATION: True,
         _KEY_SIMPLE_ENGLISH_PLURAL_RULES: False,
         _KEY_INPUT_LIMIT: 0,
+        _KEY_USE_TXT_VEC: False
     }
 
     MODEL_FILE = 'stwfsa_predictor.zip'
diff --git a/setup.py b/setup.py
index b6a6beadd..2ced56168 100644
--- a/setup.py
+++ b/setup.py
@@ -35,7 +35,7 @@ def read(fname):
         'gunicorn',
         'numpy==1.18.*',
         'optuna==2.2.0',
-        'stwfsapy==0.1.5'
+        'stwfsapy==0.2.*'
     ],
     tests_require=['py', 'pytest', 'requests'],
     extras_require={


Test patch:
diff --git a/tests/test_backend_stwfsa.py b/tests/test_backend_stwfsa.py
index 352092f11..e27f11c6b 100644
--- a/tests/test_backend_stwfsa.py
+++ b/tests/test_backend_stwfsa.py
@@ -57,6 +57,7 @@ def test_stwfsa_default_params(project):
         'expand_abbreviation_with_punctuation': True,
         'simple_english_plural_rules': False,
         'input_limit': 0,
+        'use_txt_vec': False,
     }
     actual_params = stwfsa.params
     assert expected_default_params == actual_params