Repository: piskvorky/gensim

Issue:
make `remove_stopwords()` behavior more consistent
(triggered by SO question: https://stackoverflow.com/questions/67944732/using-my-own-stopword-list-with-gensim-corpora-textcorpus-textcorpus/67951592#67951592)

Gensim has two `remove_stopwords()` functions with similar, but slightly-different behavior that risks confusing users. 

`gensim.parsing.preprocessing.remove_stopwords` takes a space-delimited string, and always consults the *current* value of `gensim.parsing.preprocessing.STOPWORDS` (including any user reassignments). 

By contrast, `gensim.corpora.textcorpus.remove_stopwords` takes a list-of-tokens, allows the specification of an alternate list-of-stopwords, but if allowed to use its 'default' stopwords, uses only the value of `gensim.parsing.preprocessing.STOPWORDS` that was captured when the function was defined (which could miss later user redefinitions).

Avoiding the reuse of identical function names that take different argument-types would help avoid user/code-reviewer confusion. Also, capturing the value of a variable that might change leads to confusing function-behavior: the token version could just as easily consult `gensim.parsing.preprocessing.STOPWORDS` on each call (as the string version already does). 


Discussion:
Hi @gojomo, I would like to work on this issue.
I need some clarification regarding this - 
> Avoiding the reuse of identical function names that take different argument-types would help avoid user/code-reviewer confusion.

Should we keep two different names for the above two functions? For example, we can keep the name of the function in `gensim.corpora.textcorpus` as **remove_stopwords_token()** or something more understandable to the user. 
Or
Taking the `gensim.parsing.preprocessing.remove_stopwords` as standard and using it in the `gensim.corpora.textcorpus.TextCorpus` for preprocessing.

Would like to know your thoughts on the above two or any other function name you have in mind.

It'd be good to have @piskvorky's preferences here, but my rough approach might be:

* see which one is more used in more places throughout our code - let that one keep its existing name/signature (principle of: cause minimal adjustment pain for others) - I suspect that will be the `preprocessing` one, as that *sounds* more generic/central.
* try to redefine the other one to be more clearly named, and be clearly related to the other - perhaps by relocating into the same module as the other (directly alongside it in the source code), and calling/being-called-by the other as appropriate. If it's the list-of-tokens one that gets renamed, I'd find `remove_stopword_tokens()` more naturally descriptive in an imperative language style, as in "**remove** the **stopword tokens**"". 
* ensure both respect the global `STOPWORDS` (if the user has redefined it), and both offer similar ability to specify alternate stopwords
* wrap the old function that's been removed with a deprecation warning pointing to the new alternative, so any changes don't immediately break old code, but that someday the less-preferable, confusing, older function can be removed entirely. at the same time, ensure our code uses the preferred, not the deprecated, function.

Thanks for your interest!

I wasn't aware of `gensim.corpora.textcorpus.remove_stopwords`, which seems to have been added in https://github.com/RaRe-Technologies/gensim/pull/1459.

Yes, IMO such functions should live in `gensim.preprocessing` – and better yet, outside of Gensim. I'd prefer Gensim to be pre-processing free, for reasons stated in the tutorials:

1) (primarily) Preprocessing is domain-specific and, similarly to evaluation, works best when tuned to your specific application. I don't have much faith in general-purpose NLP.

2) (secondary) There are other, more focused NLP libraries for text preprocessing. (But see my caveat above)

For now, consolidating preprocessing under the `preprocessing` module is preferable, thanks.
Thanks, @gojomo, and @piskvorky for this detailed explanation.
>For now, consolidating preprocessing under the preprocessing module is preferable, thanks.

So, I will rename the function in `gensim.corpora.textcorpus` ( as it's not used in any other module) and will put that under the `preprocessing` module.

There are other generic preprocessing function in `gensim.corpora.textcorpus` ( [remove_short()](https://github.com/RaRe-Technologies/gensim/blob/develop/gensim/corpora/textcorpus.py#L72), [lower_to_unicode()](https://github.com/RaRe-Technologies/gensim/blob/develop/gensim/corpora/textcorpus.py#L91), [strip_multiple_whitespaces()](https://github.com/RaRe-Technologies/gensim/blob/develop/gensim/corpora/textcorpus.py#L117) ). Should I also move those functions and similar preprocessing functions present in other modules into  the `preprocessing` module.
Yes. Such one-liners are weird, but if they must exist, let them exist under `preprocessing`.
ok, I will make those changes and link a PR soon.
Once it's working in its new location, I think we may then see some further ideas on clearer naming. 

@piskvorky - do you suggest  the 1-liners that move into `.preprocessing.` need deprecated aliases passing through from their old names in `.textcorpus.`?
Is an alias even needed? `from gensim.preprocessing import X, Y, Z` ought to do the same job.
If they all keep the same names, sure. But if usage from the disfavored place is to be discouraged and even eventually removed, aliases would be required to attach the deprecation warnings.
>But if usage from the disfavored place is to be discouraged and even eventually removed, aliases would be required to attach the deprecation warnings.

In that case, should I keep the function names in `.preprocessing` **same as** in `.textcorpus`? 
Also there are some functions, like, `strip_multiple_whitespaces()` which exist in both [textcorpus](https://github.com/RaRe-Technologies/gensim/blob/develop/gensim/corpora/textcorpus.py#L117) and [preprocessing](https://github.com/RaRe-Technologies/gensim/blob/develop/gensim/parsing/preprocessing.py#L232) module with same name and functionality.

Currently, I am doing this -
```py
import gensim.parsing.preprocessing as preprocessing

@deprecated("Function will be removed in future releases,use gensim.parsing.preprocessing.strip_multiple_whitespaces() instead")
def strip_multiple_whitespaces(s):
    return preprocessing.strip_multiple_whitespaces(s)
```
If the functions are truly identical in signature & functionality, I think the approach you've shown, with a deprecating-forwarding-function, is the diligent incremental improvement that's sure to warn, rather than break, any uses of the `.textcorpus.` versions. (Of course, there might be zero users-in-the-field calling those. It's unfortunate the duplicate behavior slipped in, making more work now to gracefully remove it.)
I don't get it. If the signature & functionality is the same, what's the deprecation for? No code will break if you just import the names from their location, so why bother users with some warnings?
If it was an error to have duplicated (or misplaced) such functions, I'd have thought that we'd want to eventually fix the error, and shrink the module where they don't belong. 

It's a 'code smell' to leave such redundant cruft in place indefinitely, making source files & API docs larger without tangible benefit, and at a cost in code readability. There might well be zero outside users who are calling the misplaced `.textcorpus.` versions... but if they're left in place as options with no discouragement, future users might mix-and-match calls, multiplying the prospect future code readers have to wonder, & then research: "is there a reason for, and difference between, these two modules' similar-looking functions?"

"There should be one— and preferably only one —obvious way to do it."

Personally, I'd just remove the redundant versions ASAP – "now is better than never" – add a release note, and let people who hit an error due to the change (if any such people exist) fix their code.
I still don't get it. TextCorpus needs to import these functions anyway, because its code makes use of them.

So the imports will be there either way. My question was, why re-wrap them in a deprecation? How does that help us maintainers, or the users?

Or are you suggesting removing the one-liners altogether (= rewriting TextCorpus), instead of moving them to `preprocessing`?
>So the imports will be there either way.

Yes, we need to imports those functions in Textcorpus thus any `user code` using the previous implementation will not break. Technically it will be correct and users will not face any issue.

According to @gojomo and what I have understood (please correct me if I am wrong) -
If we let users call those functions directly from textcorpus (like `textcorpus.strip_multiple_whitespaces()`) without any discouragement (by not giving any deprecating warning for such calls), it might create some sort of confusions for them and future users (who will see both the calls `textcorpus.strip_multiple_whitespaces()` & `preprocessing.strip_multiple_whitespaces()` across different codes).

But as there would be very few users who are using `gensim` preprocessing functions directly (as libraries like `nltk` are more preferable for such tasks), deprecation warnings may not provide much benefit. 

> I still don't get it. TextCorpus needs to import these functions anyway, because its code makes use of them.
> 
> So the imports will be there either way. My question was, why re-wrap them in a deprecation? How does that help us maintainers, or the users?
> 
> Or are you suggesting removing the one-liners altogether (= rewriting TextCorpus), instead of moving them to `preprocessing`?

Aha. OK, **if** the functions keep the same name & merely move to a more-sensible place (`preprocessing`), then I'm overthinking this, since anywhere that imports a function also unavoidably exports it. (Though then, without appearing in the module docs.)

But the concern remains with the `textcorpus.remove_stopwords()` function itself. If it both *moves* and its name *changes* for clarity - which I think is a good idea, given the potential confusion with `preprocessing.remove_stopwords()` – should `textcorpus` still essentially provide a `remove_stopwords` function with the old signature/behavior, or not? And if it does, is there a warning that it is superfluous & deprecated, or does it remain as a monument to a past error forever? For example, would the following line in `textcorpus.py` be a service or disservice to Gensim users?

```
from gensim.preprocessing. import remove_stopword_tokens as remove_stopwords
```

(And, the concern might repeat if any other `textcorpus` utility preprocessing functions, when moved to `preprocessing`, deserve any other naming/parameter changes to be more clearly consistent with other functions there.)
Yeah, I don't think anyone's using `textcorpus.remove_tokens`. And if they do, the update to their code should be trivial.

So let's do whatever makes sense to us. IMO that means giving the functions suitable names and moving them under `preprocessing`.

Resolving patch:
diff --git a/gensim/corpora/lowcorpus.py b/gensim/corpora/lowcorpus.py
--- a/gensim/corpora/lowcorpus.py
+++ b/gensim/corpora/lowcorpus.py
@@ -11,28 +11,11 @@
 
 from gensim import utils
 from gensim.corpora import IndexedCorpus
-
+from gensim.parsing.preprocessing import split_on_space
 
 logger = logging.getLogger(__name__)
 
 
-def split_on_space(s):
-    """Split line by spaces, used in :class:`gensim.corpora.lowcorpus.LowCorpus`.
-
-    Parameters
-    ----------
-    s : str
-        Some line.
-
-    Returns
-    -------
-    list of str
-        List of tokens from `s`.
-
-    """
-    return [word for word in utils.to_unicode(s).strip().split(' ') if word]
-
-
 class LowCorpus(IndexedCorpus):
     """Corpus handles input in `GibbsLda++ format <http://gibbslda.sourceforge.net/>`_.
 
@@ -86,7 +69,7 @@ def __init__(self, fname, id2word=None, line2words=split_on_space):
             If not provided, the mapping is constructed directly from `fname`.
         line2words : callable, optional
             Function which converts lines(str) into tokens(list of str),
-            using :func:`~gensim.corpora.lowcorpus.split_on_space` as default.
+            using :func:`~gensim.parsing.preprocessing.split_on_space` as default.
 
         """
         IndexedCorpus.__init__(self, fname)
diff --git a/gensim/corpora/textcorpus.py b/gensim/corpora/textcorpus.py
--- a/gensim/corpora/textcorpus.py
+++ b/gensim/corpora/textcorpus.py
@@ -44,93 +44,15 @@
 
 from gensim import interfaces, utils
 from gensim.corpora.dictionary import Dictionary
-from gensim.parsing.preprocessing import STOPWORDS, RE_WHITESPACE
+from gensim.parsing.preprocessing import (
+    remove_stopword_tokens, remove_short_tokens,
+    lower_to_unicode, strip_multiple_whitespaces,
+)
 from gensim.utils import deaccent, simple_tokenize
 
 logger = logging.getLogger(__name__)
 
 
-def remove_stopwords(tokens, stopwords=STOPWORDS):
-    """Remove stopwords using list from `gensim.parsing.preprocessing.STOPWORDS`.
-
-    Parameters
-    ----------
-    tokens : iterable of str
-        Sequence of tokens.
-    stopwords : iterable of str, optional
-        Sequence of stopwords
-
-    Returns
-    -------
-    list of str
-        List of tokens without `stopwords`.
-
-    """
-    return [token for token in tokens if token not in stopwords]
-
-
-def remove_short(tokens, minsize=3):
-    """Remove tokens shorter than `minsize` chars.
-
-    Parameters
-    ----------
-    tokens : iterable of str
-        Sequence of tokens.
-    minsize : int, optimal
-        Minimal length of token (include).
-
-    Returns
-    -------
-    list of str
-        List of tokens without short tokens.
-
-    """
-    return [token for token in tokens if len(token) >= minsize]
-
-
-def lower_to_unicode(text, encoding='utf8', errors='strict'):
-    """Lowercase `text` and convert to unicode, using :func:`gensim.utils.any2unicode`.
-
-    Parameters
-    ----------
-    text : str
-        Input text.
-    encoding : str, optional
-        Encoding that will be used for conversion.
-    errors : str, optional
-        Error handling behaviour, used as parameter for `unicode` function (python2 only).
-
-    Returns
-    -------
-    str
-        Unicode version of `text`.
-
-    See Also
-    --------
-    :func:`gensim.utils.any2unicode`
-        Convert any string to unicode-string.
-
-    """
-    return utils.to_unicode(text.lower(), encoding, errors)
-
-
-def strip_multiple_whitespaces(s):
-    """Collapse multiple whitespace characters into a single space.
-
-    Parameters
-    ----------
-    s : str
-        Input string
-
-    Returns
-    -------
-    str
-        String with collapsed whitespaces.
-
-    """
-    return RE_WHITESPACE.sub(" ", s)
-
-
 class TextCorpus(interfaces.CorpusABC):
     """Helper class to simplify the pipeline of getting BoW vectors from plain text.
 
@@ -177,12 +99,12 @@ class TextCorpus(interfaces.CorpusABC):
 
     The default preprocessing consists of:
 
-    #. :func:`~gensim.corpora.textcorpus.lower_to_unicode` - lowercase and convert to unicode (assumes utf8 encoding)
+    #. :func:`~gensim.parsing.preprocessing.lower_to_unicode` - lowercase and convert to unicode (assumes utf8 encoding)
     #. :func:`~gensim.utils.deaccent`- deaccent (asciifolding)
-    #. :func:`~gensim.corpora.textcorpus.strip_multiple_whitespaces` - collapse multiple whitespaces into a single one
+    #. :func:`~gensim.parsing.preprocessing.strip_multiple_whitespaces` - collapse multiple whitespaces into one
     #. :func:`~gensim.utils.simple_tokenize` - tokenize by splitting on whitespace
-    #. :func:`~gensim.corpora.textcorpus.remove_short` - remove words less than 3 characters long
-    #. :func:`~gensim.corpora.textcorpus.remove_stopwords` - remove stopwords
+    #. :func:`~gensim.parsing.preprocessing.remove_short_tokens` - remove words less than 3 characters long
+    #. :func:`~gensim.parsing.preprocessing.remove_stopword_tokens` - remove stopwords
 
     """
 
@@ -204,15 +126,15 @@ def __init__(self, input=None, dictionary=None, metadata=False, character_filter
             Each will be applied to the text of each document in order, and should return a single string with
             the modified text. For Python 2, the original text will not be unicode, so it may be useful to
             convert to unicode as the first character filter.
-            If None - using :func:`~gensim.corpora.textcorpus.lower_to_unicode`,
-            :func:`~gensim.utils.deaccent` and :func:`~gensim.corpora.textcorpus.strip_multiple_whitespaces`.
+            If None - using :func:`~gensim.parsing.preprocessing.lower_to_unicode`,
+            :func:`~gensim.utils.deaccent` and :func:`~gensim.parsing.preprocessing.strip_multiple_whitespaces`.
         tokenizer : callable, optional
             Tokenizer for document, if None - using :func:`~gensim.utils.simple_tokenize`.
         token_filters : iterable of callable, optional
             Each will be applied to the iterable of tokens in order, and should return another iterable of tokens.
             These filters can add, remove, or replace tokens, or do nothing at all.
-            If None - using :func:`~gensim.corpora.textcorpus.remove_short` and
-            :func:`~gensim.corpora.textcorpus.remove_stopwords`.
+            If None - using :func:`~gensim.parsing.preprocessing.remove_short_tokens` and
+            :func:`~gensim.parsing.preprocessing.remove_stopword_tokens`.
 
         Examples
         --------
@@ -254,7 +176,7 @@ def __init__(self, input=None, dictionary=None, metadata=False, character_filter
 
         self.token_filters = token_filters
         if self.token_filters is None:
-            self.token_filters = [remove_short, remove_stopwords]
+            self.token_filters = [remove_short_tokens, remove_stopword_tokens]
 
         self.length = None
         self.dictionary = None
diff --git a/gensim/parsing/preprocessing.py b/gensim/parsing/preprocessing.py
--- a/gensim/parsing/preprocessing.py
+++ b/gensim/parsing/preprocessing.py
@@ -68,17 +68,20 @@
 RE_WHITESPACE = re.compile(r"(\s)+", re.UNICODE)
 
 
-def remove_stopwords(s):
+def remove_stopwords(s, stopwords=None):
     """Remove :const:`~gensim.parsing.preprocessing.STOPWORDS` from `s`.
 
     Parameters
     ----------
     s : str
+    stopwords : iterable of str, optional
+        Sequence of stopwords
+        If None - using :const:`~gensim.parsing.preprocessing.STOPWORDS`
 
     Returns
     -------
     str
-        Unicode string without :const:`~gensim.parsing.preprocessing.STOPWORDS`.
+        Unicode string without `stopwords`.
 
     Examples
     --------
@@ -90,7 +93,29 @@ def remove_stopwords(s):
 
     """
     s = utils.to_unicode(s)
-    return " ".join(w for w in s.split() if w not in STOPWORDS)
+    return " ".join(remove_stopword_tokens(s.split(), stopwords))
+
+
+def remove_stopword_tokens(tokens, stopwords=None):
+    """Remove stopword tokens using list `stopwords`.
+
+    Parameters
+    ----------
+    tokens : iterable of str
+        Sequence of tokens.
+    stopwords : iterable of str, optional
+        Sequence of stopwords
+        If None - using :const:`~gensim.parsing.preprocessing.STOPWORDS`
+
+    Returns
+    -------
+    list of str
+        List of tokens without `stopwords`.
+
+    """
+    if stopwords is None:
+        stopwords = STOPWORDS
+    return [token for token in tokens if token not in stopwords]
 
 
 def strip_punctuation(s):
@@ -170,7 +195,26 @@ def strip_short(s, minsize=3):
 
     """
     s = utils.to_unicode(s)
-    return " ".join(e for e in s.split() if len(e) >= minsize)
+    return " ".join(remove_short_tokens(s.split(), minsize))
+
+
+def remove_short_tokens(tokens, minsize=3):
+    """Remove tokens shorter than `minsize` chars.
+
+    Parameters
+    ----------
+    tokens : iterable of str
+        Sequence of tokens.
+    minsize : int, optimal
+        Minimal length of token (include).
+
+    Returns
+    -------
+    list of str
+        List of tokens without short tokens.
+    """
+
+    return [token for token in tokens if len(token) >= minsize]
 
 
 def strip_numeric(s):
@@ -308,6 +352,49 @@ def stem_text(text):
 stem = stem_text
 
 
+def lower_to_unicode(text, encoding='utf8', errors='strict'):
+    """Lowercase `text` and convert to unicode, using :func:`gensim.utils.any2unicode`.
+
+    Parameters
+    ----------
+    text : str
+        Input text.
+    encoding : str, optional
+        Encoding that will be used for conversion.
+    errors : str, optional
+        Error handling behaviour, used as parameter for `unicode` function (python2 only).
+
+    Returns
+    -------
+    str
+        Unicode version of `text`.
+
+    See Also
+    --------
+    :func:`gensim.utils.any2unicode`
+        Convert any string to unicode-string.
+
+    """
+    return utils.to_unicode(text.lower(), encoding, errors)
+
+
+def split_on_space(s):
+    """Split line by spaces, used in :class:`gensim.corpora.lowcorpus.LowCorpus`.
+
+    Parameters
+    ----------
+    s : str
+        Some line.
+
+    Returns
+    -------
+    list of str
+        List of tokens from `s`.
+
+    """
+    return [word for word in utils.to_unicode(s).strip().split(' ') if word]
+
+
 DEFAULT_FILTERS = [
     lambda x: x.lower(), strip_tags, strip_punctuation,
     strip_multiple_whitespaces, strip_numeric,


Test patch:
diff --git a/gensim/test/test_parsing.py b/gensim/test/test_parsing.py
--- a/gensim/test/test_parsing.py
+++ b/gensim/test/test_parsing.py
@@ -7,12 +7,17 @@
 
 import logging
 import unittest
+
+import mock
 import numpy as np
 
 from gensim.parsing.preprocessing import (
+    remove_short_tokens,
+    remove_stopword_tokens,
     remove_stopwords,
     stem_text,
     split_alphanum,
+    split_on_space,
     strip_multiple_whitespaces,
     strip_non_alphanum,
     strip_numeric,
@@ -21,7 +26,6 @@
     strip_tags,
 )
 
-
 # several documents
 doc1 = """C'est un trou de verdure où chante une rivière,
 Accrochant follement aux herbes des haillons
@@ -76,6 +80,26 @@ def test_split_alphanum(self):
     def test_strip_stopwords(self):
         self.assertEqual(remove_stopwords("the world is square"), "world square")
 
+        # confirm redifining the global `STOPWORDS` working
+        with mock.patch('gensim.parsing.preprocessing.STOPWORDS', frozenset(["the"])):
+            self.assertEqual(remove_stopwords("the world is square"), "world is square")
+
+    def test_strip_stopword_tokens(self):
+        self.assertEqual(remove_stopword_tokens(["the", "world", "is", "sphere"]), ["world", "sphere"])
+
+        # confirm redifining the global `STOPWORDS` working
+        with mock.patch('gensim.parsing.preprocessing.STOPWORDS', frozenset(["the"])):
+            self.assertEqual(
+                remove_stopword_tokens(["the", "world", "is", "sphere"]),
+                ["world", "is", "sphere"]
+            )
+
+    def test_strip_short_tokens(self):
+        self.assertEqual(remove_short_tokens(["salut", "les", "amis", "du", "59"], 3), ["salut", "les", "amis"])
+
+    def test_split_on_space(self):
+        self.assertEqual(split_on_space(" salut   les  amis du 59 "), ["salut", "les", "amis", "du", "59"])
+
     def test_stem_text(self):
         target = \
             "while it is quit us to be abl to search a larg " + \