Repository: amueller/word_cloud

Issue:
Remove stopwords after creating bigrams to prevent incorrect bigrams
#### Description
I am dealing with a corpus containing the phrase "thank you very much" frequently.  Currently this results in a frequent bigram "thank much".  This results from generating the bigrams without a concept of removed stopwords.

#### Steps/Code to Reproduce
Create a word cloud containing frequent phrases having internal stopwords.

#### Expected Results
I expected one of two things.  Either that 1) "thank you", "very much" would be frequent bigrams (but not "you very", which consists entirely of bigrams.)  Or that 2) "thank" and "much" would show up as frequent unigrams.  I think the first option is the more elegant solution.

#### Actual Results
"thank much" was a frequent bigram.

#### Versions
```
>>> import wordcloud; print("wordcoud", wordcloud.__version__)
wordcoud 1.6.0
```


Discussion:
carlgieringer: Weird, the tests pass for me locally.  Why do you think that this `test_small_canvas` would fail remotely?  Here is my system info:

```
$ python
Python 3.7.6 (default, Feb 22 2020, 21:07:37)
[Clang 11.0.0 (clang-1100.0.33.17)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import platform; print(platform.platform())
Darwin-19.3.0-x86_64-i386-64bit
>>> import sys; print("Python", sys.version)
Python 3.7.6 (default, Feb 22 2020, 21:07:37)
[Clang 11.0.0 (clang-1100.0.33.17)]
>>> import numpy; print("NumPy", numpy.__version__)
NumPy 1.18.1
>>> import matplotlib; print("matplotlib", matplotlib.__version__)
matplotlib 3.1.3
>>> import wordcloud; print("wordcoud", wordcloud.__version__)
wordcoud 1.6.0.post28+gf26dd63
```
amueller: Hey. Thank you for the contribution!
There was a recent fix in the small canvas test here: #525.
Can you make sure you have the current version of master? Sorry I'm a bit swamped but I'll try to review this soon.
carlgieringer: Yes, my branch is up-to-date with your master, including #525.
carlgieringer: Weird.  Error looks unrelated to my changes:

```
BUILD FAILED (OS X 10.13.6 using python-build 20180424)

Inspect or clean up the working tree at /var/folders/nz/vv4_9tw56nv9k3tkvyszvwg80000gn/T/python-build.20200301213032.36590

Results logged to /var/folders/nz/vv4_9tw56nv9k3tkvyszvwg80000gn/T/python-build.20200301213032.36590.log

Last 10 log lines:

(cd /Users/travis/.pyenv/versions/3.4.8/share/man/man1; ln -s python3.4.1 python3.1)

if test "xupgrade" != "xno"  ; then \

		case upgrade in \

			upgrade) ensurepip="--upgrade" ;; \

			install|*) ensurepip="" ;; \

		esac; \

		 ./python.exe -E -m ensurepip \

			$ensurepip --root=/ ; \

	fi

Ignoring ensurepip failure: pip 9.0.1 requires SSL/TLS

Traceback (most recent call last):

  File "../addons/travis/install_pyenv.py", line 132, in <module>

    install(os.environ['PYTHON_VERSION'])

  File "../addons/travis/install_pyenv.py", line 121, in install

    check_output(cmd, shell=True)

  File "/usr/local/Cellar/python@2/2.7.17/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 223, in check_output

    raise CalledProcessError(retcode, cmd, output=output)

subprocess.CalledProcessError: Command 'eval "$( pyenv init - )"

pyenv install 3.4.8' returned non-zero exit status 1

A command failed while executing BEFORE_INSTALL step.

  Return code:

    1

  Command:

    python ../addons/travis/install_pyenv.py

  Working directory:

    /Users/travis/build/amueller/word_cloud
```
amueller: Thanks for the update. I think the travis failure might be unrelated, but I have to double check. The diff seems like it contains unrelated content, though?
amueller: yeah the travis failure is unrelated. I should really update that....

Patch:
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
new file mode 100644
index 0000000..2dd0a75
--- /dev/null
+++ b/CONTRIBUTING.md
@@ -0,0 +1,46 @@
+# Contributing
+
+To contribute to wordcloud, you'll need to follow the instructions in
+[Creating a pull request from a fork](https://help.github.com/en/github/collaborating-with-issues-and-pull-requests/creating-a-pull-request-from-a-fork).
+
+In addition to the general procedure for creating a pull request, please follow
+the following steps:
+
+## Before starting development
+
+### Use a correct version of Python
+
+Python 3.7.x should be fine for development.
+
+```
+python --version
+> Python 3.7.6
+```
+
+### Install all dependencies
+
+```
+pip install -U -r requirements.txt -r requirements-dev.txt
+```
+
+### Ensure that files are correctly formatted
+
+```
+flake8
+```
+
+### Ensure that tests pass
+
+```
+pip install -e .
+pytest
+```
+
+## Before creating a pull request
+
+### Confirm formatting and test passage
+
+```
+flake8
+pytest
+```
diff --git a/wordcloud/tokenization.py b/wordcloud/tokenization.py
index b9c28b3..57e4e22 100644
--- a/wordcloud/tokenization.py
+++ b/wordcloud/tokenization.py
@@ -36,26 +36,27 @@ def pairwise(iterable):
     return zip(a, b)
 
 
-def unigrams_and_bigrams(words, normalize_plurals=True):
+def unigrams_and_bigrams(words, stopwords, normalize_plurals=True, collocation_threshold=30):
+    # We must create the bigrams before removing the stopword tokens from the words, or else we get bigrams like
+    # "thank much" from "thank you very much".  But bigrams consisting entirely of stopwords are still invalid.
+    bigrams = list(p for p in pairwise(words) if not all(w in stopwords for w in p))
     n_words = len(words)
-    # make tuples of two words following each other
-    bigrams = list(pairwise(words))
     counts_unigrams, standard_form = process_tokens(
         words, normalize_plurals=normalize_plurals)
     counts_bigrams, standard_form_bigrams = process_tokens(
         [" ".join(bigram) for bigram in bigrams],
         normalize_plurals=normalize_plurals)
     # create a copy of counts_unigram so the score computation is not changed
-    counts = counts_unigrams.copy()
+    orig_counts = counts_unigrams.copy()
 
-    # decount words inside bigrams
+    # Include bigrams that are also collocations
     for bigram_string, count in counts_bigrams.items():
         bigram = tuple(bigram_string.split(" "))
-        # collocation detection (30 is arbitrary):
         word1 = standard_form[bigram[0].lower()]
         word2 = standard_form[bigram[1].lower()]
 
-        if score(count, counts[word1], counts[word2], n_words) > 30:
+        collocation_score = score(count, orig_counts[word1], orig_counts[word2], n_words)
+        if collocation_score > collocation_threshold:
             # bigram is a collocation
             # discount words in unigrams dict. hack because one word might
             # appear in multiple collocations at the same time
@@ -63,10 +64,9 @@ def unigrams_and_bigrams(words, normalize_plurals=True):
             counts_unigrams[word1] -= counts_bigrams[bigram_string]
             counts_unigrams[word2] -= counts_bigrams[bigram_string]
             counts_unigrams[bigram_string] = counts_bigrams[bigram_string]
-    words = list(counts_unigrams.keys())
-    for word in words:
-        # remove empty / negative counts
-        if counts_unigrams[word] <= 0:
+    for word, count in list(counts_unigrams.items()):
+        # remove stopwords and empty / negative counts
+        if all(w in stopwords for w in word.split(" ")) or count <= 0:
             del counts_unigrams[word]
     return counts_unigrams
 
diff --git a/wordcloud/wordcloud.py b/wordcloud/wordcloud.py
index a026bb6..88188fd 100644
--- a/wordcloud/wordcloud.py
+++ b/wordcloud/wordcloud.py
@@ -1,3 +1,4 @@
+# coding=utf-8
 # Author: Andreas Christian Mueller <t3kcit@gmail.com>
 #
 # (c) 2012
@@ -273,6 +274,14 @@ class WordCloud(object):
     min_word_length : int, default=0
         Minimum number of letters a word must have to be included.
 
+    collocation_threshold: int, default=30
+        Bigrams must have a Dunning likelihood collocation score greater than this
+        parameter to be counted as bigrams. Default of 30 is arbitrary.
+
+        See Manning, C.D., Manning, C.D. and Schütze, H., 1999. Foundations of
+        Statistical Natural Language Processing. MIT press, p. 162
+        https://nlp.stanford.edu/fsnlp/promo/colloc.pdf#page=22
+
     Attributes
     ----------
     ``words_`` : dict of string to float
@@ -303,7 +312,7 @@ class WordCloud(object):
                  relative_scaling='auto', regexp=None, collocations=True,
                  colormap=None, normalize_plurals=True, contour_width=0,
                  contour_color='black', repeat=False,
-                 include_numbers=False, min_word_length=0):
+                 include_numbers=False, min_word_length=0, collocation_threshold=30):
         if font_path is None:
             font_path = FONT_PATH
         if color_func is None and colormap is None:
@@ -354,6 +363,7 @@ class WordCloud(object):
         self.repeat = repeat
         self.include_numbers = include_numbers
         self.min_word_length = min_word_length
+        self.collocation_threshold = collocation_threshold
 
     def fit_words(self, frequencies):
         """Create a word_cloud from words and frequencies.
@@ -557,15 +567,11 @@ class WordCloud(object):
         include all those things.
         """
 
-        stopwords = set([i.lower() for i in self.stopwords])
-
         flags = (re.UNICODE if sys.version < '3' and type(text) is unicode  # noqa: F821
                  else 0)
         regexp = self.regexp if self.regexp is not None else r"\w[\w']+"
 
         words = re.findall(regexp, text, flags)
-        # remove stopwords
-        words = [word for word in words if word.lower() not in stopwords]
         # remove 's
         words = [word[:-2] if word.lower().endswith("'s") else word
                  for word in words]
@@ -576,9 +582,12 @@ class WordCloud(object):
         if self.min_word_length:
             words = [word for word in words if len(word) >= self.min_word_length]
 
+        stopwords = set([i.lower() for i in self.stopwords])
         if self.collocations:
-            word_counts = unigrams_and_bigrams(words, self.normalize_plurals)
+            word_counts = unigrams_and_bigrams(words, stopwords, self.normalize_plurals, self.collocation_threshold)
         else:
+            # remove stopwords
+            words = [word for word in words if word.lower() not in stopwords]
             word_counts, _ = process_tokens(words, self.normalize_plurals)
 
         return word_counts


Test patch:
diff --git a/test/test_wordcloud.py b/test/test_wordcloud.py
index cce524e..fca84ea 100644
--- a/test/test_wordcloud.py
+++ b/test/test_wordcloud.py
@@ -41,12 +41,23 @@ Namespaces are one honking great idea -- let's do more of those!
     46 09 55 05 82   23 17 25 35 94   08 128
 """
 
+STOPWORDED_COLLOCATIONS = """
+thank you very much
+thank you very much
+thank you very much
+thanks
+"""
+
+SMALL_CANVAS = """
+better late than never someone will say
+"""
+
 
 def test_collocations():
-    wc = WordCloud(collocations=False, stopwords=[])
+    wc = WordCloud(collocations=False, stopwords=set())
     wc.generate(THIS)
 
-    wc2 = WordCloud(collocations=True, stopwords=[])
+    wc2 = WordCloud(collocations=True, stopwords=set())
     wc2.generate(THIS)
 
     assert "is better" in wc2.words_
@@ -54,6 +65,18 @@ def test_collocations():
     assert "way may" not in wc2.words_
 
 
+def test_collocation_stopwords():
+    wc = WordCloud(collocations=True, stopwords={"you", "very"}, collocation_threshold=9)
+    wc.generate(STOPWORDED_COLLOCATIONS)
+
+    assert "thank you" in wc.words_
+    assert "very much" in wc.words_
+    # "thank" will have been removed in favor of the bigrams including "thank"
+    assert "thank" not in wc.words_
+    # a bigram of all stopwords will be removed
+    assert "you very" not in wc.words_
+
+
 def test_plurals_numbers():
     text = THIS + "\n" + "1 idea 2 ideas three ideas although many Ideas"
     wc = WordCloud(stopwords=[]).generate(text)
@@ -352,8 +375,8 @@ def test_recolor_too_small_set_default():
 def test_small_canvas():
     # check font size fallback works on small canvas
     wc = WordCloud(max_words=50, width=21, height=21)
-    wc.generate(THIS)
-    assert len(wc.layout_) == 1
+    wc.generate(SMALL_CANVAS)
+    assert len(wc.layout_) > 0
 
 
 def test_tiny_canvas():