Repository: biolab/orange3-text

Issue:
WordCloud crashes
``` text
LookupError                                   Traceback (most recent call last):
  File "/home/user/orange3/Orange/canvas/scheme/widgetsscheme.py", line 722, in process_signals_for_widget
    handler(*args)
  File "/home/user/orange3/textmining/orangecontrib/text/widgets/owwordcloud.py", line 241, in on_corpus_change
    self.corpus_counter = Counter(w for doc in data.tokens for w in doc)
  File "/home/user/orange3/textmining/orangecontrib/text/corpus.py", line 159, in tokens
    self._tokens = p(self.documents)
  File "/home/user/orange3/textmining/orangecontrib/text/preprocess.py", line 128, in __call__
    tokens.append(self.tokenizer(document))
  File "/home/user/venvs/orange3/lib/python3.4/site-packages/nltk-3.1-py3.4.egg/nltk/tokenize/__init__.py", line 104, in word_tokenize
    return [token for sent in sent_tokenize(text, language)
  File "/home/user/venvs/orange3/lib/python3.4/site-packages/nltk-3.1-py3.4.egg/nltk/tokenize/__init__.py", line 88, in sent_tokenize
    tokenizer = load('tokenizers/punkt/{0}.pickle'.format(language))
  File "/home/user/venvs/orange3/lib/python3.4/site-packages/nltk-3.1-py3.4.egg/nltk/data.py", line 796, in load
    opened_resource = _open(resource_url)
  File "/home/user/venvs/orange3/lib/python3.4/site-packages/nltk-3.1-py3.4.egg/nltk/data.py", line 914, in _open
    return find(path_, path + ['']).open()
  File "/home/user/venvs/orange3/lib/python3.4/site-packages/nltk-3.1-py3.4.egg/nltk/data.py", line 636, in find
    raise LookupError(resource_not_found)
LookupError: 
**********************************************************************
  Resource 'tokenizers/punkt/PY3/english.pickle' not found.
  Please use the NLTK Downloader to obtain the resource:  >>>
  nltk.download()
  Searched in:
    - '/home/user/nltk_data'
    - '/usr/share/nltk_data'
    - '/usr/local/share/nltk_data'
    - '/usr/lib/nltk_data'
    - '/usr/local/lib/nltk_data'
    - ''
**********************************************************************
```

Why does it assume I have/want/need NLTK data? How is this different from the issue raised in https://github.com/biolab/orange3-text/pull/47? 



Discussion:
Crap, we forgot about that. You need NLTK data since the default NLTK tokenizer requires it. It differs from #47 since there we have to somehow install system packages. Here we just need to assure that one python script is run after installing a package. We already have this script written for travis 🙈 


Resolving patch:
diff --git a/orangecontrib/text/preprocess.py b/orangecontrib/text/preprocess.py
--- a/orangecontrib/text/preprocess.py
+++ b/orangecontrib/text/preprocess.py
@@ -1,3 +1,4 @@
+import nltk
 from nltk.corpus import stopwords
 from nltk.stem import WordNetLemmatizer
 from nltk.stem.porter import PorterStemmer
@@ -6,6 +7,10 @@
 
 from orangecontrib.text.corpus import Corpus
 
+# Ensure NLTK data is present
+NLTK_DATA = ["wordnet",  "stopwords",  "punkt"]
+nltk.download(NLTK_DATA)
+
 
 class Preprocessor:
     """A pre-processing class for the Orange text mining extension.
diff --git a/orangecontrib/text/widgets/owpreprocess.py b/orangecontrib/text/widgets/owpreprocess.py
--- a/orangecontrib/text/widgets/owpreprocess.py
+++ b/orangecontrib/text/widgets/owpreprocess.py
@@ -6,8 +6,9 @@
                          QComboBox, QPushButton, QStyle, QApplication,
                          QFileDialog, QLineEdit, QCheckBox)
 from nltk.corpus import stopwords
+from nltk.downloader import Downloader
 
-from Orange.widgets import gui, settings
+from Orange.widgets import gui, settings, widget
 from Orange.widgets.widget import OWWidget
 from orangecontrib.text.corpus import Corpus
 from orangecontrib.text.preprocess import (PorterStemmer as PS,
@@ -571,6 +572,13 @@ class OWPreprocess(OWWidget):
     persistent_data_filter = settings.Setting({})
     preprocessors = []  # Pre-processing modules for the current run.
 
+    UserAdviceMessages = [
+        widget.Message(
+            "Some preprocessing methods require data (like word relationships, stop words, "
+            "punctuation rules etc.) from the NLTK package. This data, if you didn't have it "
+            "already, was downloaded to: {}".format(Downloader().default_download_dir()),
+            "nltk_data")]
+
     def __init__(self, parent=None):
         super().__init__(parent)
 


Test patch: