internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: biolab/orange3-text

Issue:
Preprocess Text: stopwords not found
<!--
This is an issue template. Please fill in the relevant details in the
sections below.
-->

##### Text version
<!-- From menu _Options→Add-ons→Orange3-Text_ or code `orangecontrib.text.version.full_version` -->
0.3.0

##### Orange version
<!-- From menu _Help→About→Version_ or code `Orange.version.full_version` -->
3.15.dev

##### Expected behavior
Preprocess Text successfully loads NLTK data and stopwords options appear.


##### Actual behavior
Preprocess Text shows None instead for some users.


##### Steps to reproduce the behavior
Probably has sth to do with Win machines. Install a clean version on Windows and install Text from Options-Add-ons. Check Preprocess Text under Filtering section - Stopwords.


##### Additional info (worksheets, data, screenshots, ...)
I think NLTK data might be loaded, but misplaced in the wrong directory. Check if and where the data got loaded and set Preprocess Text to find the right folder.




Discussion:

Resolving patch:
diff --git a/orangecontrib/text/preprocess/filter.py b/orangecontrib/text/preprocess/filter.py
--- a/orangecontrib/text/preprocess/filter.py
+++ b/orangecontrib/text/preprocess/filter.py
@@ -61,20 +61,24 @@ def from_file(self, path):
         # No encoding worked, raise
         raise UnicodeError("Couldn't determine file encoding")
 
-# get NLTK list of stopwords
-stopwords_listdir = []
-try:
-    stopwords_listdir = [file for file in os.listdir(stopwords._get_root())
-                         if file.islower()]
-except LookupError:     # when no NLTK data is available
-    pass
-
 
 class StopwordsFilter(BaseTokenFilter, WordListMixin):
     """ Remove tokens present in NLTK's language specific lists or a file. """
     name = 'Stopwords'
 
-    supported_languages = [file.capitalize() for file in stopwords_listdir]
+    @staticmethod
+    @wait_nltk_data
+    def supported_languages():
+        # get NLTK list of stopwords
+        stopwords_listdir = []
+        try:
+            stopwords_listdir = [file for file in
+                                 os.listdir(stopwords._get_root())
+                                 if file.islower()]
+        except LookupError:  # when no NLTK data is available
+            pass
+
+        return [file.capitalize() for file in stopwords_listdir]
 
     @wait_nltk_data
     def __init__(self, language='English', word_list=None):
diff --git a/orangecontrib/text/widgets/owpreprocess.py b/orangecontrib/text/widgets/owpreprocess.py
--- a/orangecontrib/text/widgets/owpreprocess.py
+++ b/orangecontrib/text/widgets/owpreprocess.py
@@ -331,7 +331,7 @@ def __init__(self, master):
         super().__init__(master)
 
         box = widgets.ComboBox(self, 'stopwords_language',
-                               items=[None] + preprocess.StopwordsFilter.supported_languages)
+                               items=[None] + preprocess.StopwordsFilter.supported_languages())
         box.currentIndexChanged.connect(self.stopwords_changed)
         self.stopwords_changed()
         self.method_layout.addWidget(box, self.STOPWORDS, 1)


Test patch:
Repository: biolab/orange3-text

Issue:
Bag of Words: crashes if < 11 tokens on the input
<!--
This is an issue template. Please fill in the relevant details in the
sections below.
-->

##### Text version
<!-- From menu _Options→Add-ons→Orange3-Text_ or code `orangecontrib.text.version.full_version` -->
0.3.0

##### Orange version
<!-- From menu _Help→About→Version_ or code `Orange.version.full_version` -->
3.14.dev

##### Expected behavior
Bag of Words doesn't crash on few tokens


##### Actual behavior
BoW crashes if less then 11 tokens on the input and Binary option selected.


##### Steps to reproduce the behavior
Corpus  - Preprocess Text (have it output less than 11 types) - Bag of Words (Binary)


##### Additional info (worksheets, data, screenshots, ...)





Discussion:

Resolving patch:
diff --git a/orangecontrib/text/vectorization/bagofwords.py b/orangecontrib/text/vectorization/bagofwords.py
--- a/orangecontrib/text/vectorization/bagofwords.py
+++ b/orangecontrib/text/vectorization/bagofwords.py
@@ -46,7 +46,8 @@ class BowVectorizer(BaseVectorizer):
 
     wlocals = OrderedDict((
         (COUNT, lambda tf: tf),
-        (BINARY, lambda tf: np.greater(tf, 0, dtype=np.int)),
+        (BINARY, lambda tf: np.greater(tf, 0, dtype=np.int) if tf.size
+                            else np.array([], dtype=np.int)),
         (SUBLINEAR, lambda tf: 1 + np.log(tf)),
     ))
 


Test patch:
Repository: biolab/orange3-text

Issue:
Guardian: Fix failing tests on Travis
<!--
This is an issue template. Please fill in the relevant details in the
sections below.
-->

##### Text version
<!-- From menu _Options→Add-ons→Orange3-Text_ or code `orangecontrib.text.version.full_version` -->
0.3.0

##### Orange version
<!-- From menu _Help→About→Version_ or code `Orange.version.full_version` -->
3.15.dev

##### Expected behavior
Tests pass.


##### Actual behavior
Guardian tests is failing.


##### Steps to reproduce the behavior



##### Additional info (worksheets, data, screenshots, ...)
Fix tests.




Discussion:
Issue is with invalid credentials. I am not sure whether to turn off tests for Guardian or we can somehow set test credentials.

Resolving patch:
diff --git a/orangecontrib/text/guardian.py b/orangecontrib/text/guardian.py
--- a/orangecontrib/text/guardian.py
+++ b/orangecontrib/text/guardian.py
@@ -155,7 +155,7 @@ def search(self, query, from_date=None, to_date=None, max_documents=None,
 
 
 if __name__ == '__main__':
-    credentials = TheGuardianCredentials('')
+    credentials = TheGuardianCredentials('test')
     print(credentials.valid)
     api = TheGuardianAPI(credentials=credentials)
     c = api.search('refugees', max_documents=10)


Test patch:
Repository: biolab/orange3-text

Issue:
TheGuardianCredentials returns wrong valid property
##### Text version
0.3.0
##### Orange version
3.14
##### Expected behavior
``` python
credentials = TheGuardianCredentials('<your-api-key>')
print(credentials.valid)
```
Should correctly return if key is valid

##### Actual behavior
If the given key exceeds the API limit `credentials.valid` still returns True




Discussion:

Resolving patch:
diff --git a/orangecontrib/text/guardian.py b/orangecontrib/text/guardian.py
--- a/orangecontrib/text/guardian.py
+++ b/orangecontrib/text/guardian.py
@@ -42,7 +42,7 @@ def __init__(self, key):
     def valid(self):
         """ Check if given API key is valid. """
         response = requests.get(BASE_URL, {'api-key': self.key})
-        return response.status_code != 403      # 403 == Forbidden
+        return response.status_code == 200
 
     def __eq__(self, other):
         return self.key == other.key


Test patch:
Repository: biolab/orange3-text

Issue:
Guardian: Fix failing tests on Travis
<!--
This is an issue template. Please fill in the relevant details in the
sections below.
-->

##### Text version
<!-- From menu _Options→Add-ons→Orange3-Text_ or code `orangecontrib.text.version.full_version` -->
0.3.0

##### Orange version
<!-- From menu _Help→About→Version_ or code `Orange.version.full_version` -->
3.15.dev

##### Expected behavior
Tests pass.


##### Actual behavior
Guardian tests is failing.


##### Steps to reproduce the behavior



##### Additional info (worksheets, data, screenshots, ...)
Fix tests.




Discussion:
Issue is with invalid credentials. I am not sure whether to turn off tests for Guardian or we can somehow set test credentials.
Problem persists. 😞 
The tests are failing due to the limit on the "test" key. The best solution would be to contact them to get a key with a higher limit. According to the [documentation](https://open-platform.theguardian.com/documentation) under Polling guide, this is something they are willing to do.
They have also cunningly refrained from posting any contact form. I suppose we could register another key with Orange mail and use it for testing. However, how would we hide it from everyone else given it is an OS project?
https://docs.travis-ci.com/user/environment-variables/#Defining-Variables-in-Repository-Settings

Resolving patch:
diff --git a/orangecontrib/text/guardian.py b/orangecontrib/text/guardian.py
--- a/orangecontrib/text/guardian.py
+++ b/orangecontrib/text/guardian.py
@@ -18,6 +18,7 @@
 import requests
 import math
 import json
+import os
 
 from Orange import data
 
@@ -155,7 +156,8 @@ def search(self, query, from_date=None, to_date=None, max_documents=None,
 
 
 if __name__ == '__main__':
-    credentials = TheGuardianCredentials('test')
+    key = os.getenv('THE_GUARDIAN_API_KEY', 'test')
+    credentials = TheGuardianCredentials(key)
     print(credentials.valid)
     api = TheGuardianAPI(credentials=credentials)
     c = api.search('refugees', max_documents=10)


Test patch:
diff --git a/orangecontrib/text/tests/test_guardian.py b/orangecontrib/text/tests/test_guardian.py
--- a/orangecontrib/text/tests/test_guardian.py
+++ b/orangecontrib/text/tests/test_guardian.py
@@ -1,11 +1,12 @@
 import unittest
+import os
 
 from datetime import date, datetime
 
 from orangecontrib.text import guardian
 
 
-API_KEY = 'test'
+API_KEY = os.getenv('THE_GUARDIAN_API_KEY', 'test')
 
 
 class TestCredentials(unittest.TestCase):
Repository: biolab/orange3-text

Issue:
Tweet Profiler source available?
<!--
This is an issue template. Please fill in the relevant details in the
sections below.
-->

##### Text version

0.3.0
<!-- From menu _Options→Add-ons→Orange3-Text_ or code `orangecontrib.text.version.full_version` -->


##### Orange version
<!-- From menu _Help→About→Version_ or code `Orange.version.full_version` -->
3.13

##### Additional info (worksheets, data, screenshots, ...)

Is the server code for the Tweet Profiler available somewhere? I just started working with the text addon recently and wanted to read how you implemented it and found out that it runs on a server.


Discussion:
@nikicc I don't think this is available, right?
It's not, unfortunately.

Resolving patch:
diff --git a/doc/conf.py b/doc/conf.py
--- a/doc/conf.py
+++ b/doc/conf.py
@@ -52,7 +52,7 @@
 
 # General information about the project.
 project = 'Orange3 Text Mining'
-copyright = '2015, Laboratory of Bioinformatics, Faculty of Computer Science, University of Ljubljana'
+copyright = '2018, Laboratory of Bioinformatics, Faculty of Computer Science, University of Ljubljana'
 
 # The version info for the project you're documenting, acts as replacement for
 # |version| and |release|, also used in various other places throughout the
@@ -106,7 +106,7 @@
 
 # The theme to use for HTML and HTML Help pages.  See the documentation for
 # a list of builtin themes.
-html_theme = 'nature'
+html_theme = 'alabaster'
 
 # Theme options are theme-specific and customize the look and feel of a theme
 # further.  For a list of options available for each theme, see the
@@ -268,3 +268,6 @@
 # Mapping for external documentation in Sphinx.
 #intersphinx_mapping = {'orange3': ('../../orange3/doc/visual-programming/build/html',
 #                                   '../../orange3/doc/visual-programming/build/html/objects.inv')}
+
+def setup(app):
+    app.add_stylesheet('style.css')


Test patch:
Repository: biolab/orange3-text

Issue:
Word Cloud: scale visualization
<!--
This is an issue template. Please fill in the relevant details in the
sections below.
-->

##### Text version
<!-- From menu _Options→Add-ons→Orange3-Text_ or code `orangecontrib.text.version.full_version` -->
0.3.0

##### Orange version
<!-- From menu _Help→About→Version_ or code `Orange.version.full_version` -->
3.14.dev

##### Expected behavior
Word Cloud scales visualization once everything is displayed in the window.


##### Actual behavior
Word Cloud stays small, only white space increases.


##### Steps to reproduce the behavior
![wordcloudsize](https://user-images.githubusercontent.com/12524972/41597816-0bade9c8-73be-11e8-8525-8cf9870f5562.png)



##### Additional info (worksheets, data, screenshots, ...)





Discussion:

Resolving patch:
diff --git a/orangecontrib/text/widgets/owwordcloud.py b/orangecontrib/text/widgets/owwordcloud.py
--- a/orangecontrib/text/widgets/owwordcloud.py
+++ b/orangecontrib/text/widgets/owwordcloud.py
@@ -169,6 +169,23 @@ def on_cloud_pref_change(self):
         self.webview.evalJS('OPTIONS["minRotation"] = {}; \
                              OPTIONS["maxRotation"] = {};'.format(-tilt_amount, tilt_amount))
         self.webview.evalJS('OPTIONS["rotateRatio"] = {};'.format(tilt_ratio))
+        self.webview.evalJS('''
+        OPTIONS["gridSize"] = function () {
+          return Math.round( 
+            Math.min(
+              document.getElementById("canvas").clientWidth,
+              document.getElementById("canvas").clientHeight
+            ) / 48
+          );
+        };''')
+        self.webview.evalJS('''
+        OPTIONS["weightFactor"] = function (size) {
+          return size * 
+            Math.min(
+              document.getElementById("canvas").clientWidth,
+              document.getElementById("canvas").clientHeight
+            ) / 512;
+        };''')
         # Trigger cloud redrawing by constructing new webview, because everything else fail Macintosh
         self.webview.evalJS('OPTIONS["list"] = {};'.format(self.wordlist))
         self.webview.evalJS('redrawWordCloud();')


Test patch:
Repository: biolab/orange3-text

Issue:
Twitter: cannot retrieve authors
##### Issue
#383 broke the retrieval by Author.

#### Expected behaviour

Twitter should be able to retrieve Author timelines, which apparently don't have the full_text option.

#### Error

`AttributeError: 'Status' object has no attribute 'full_text'`


Discussion:

Resolving patch:
diff --git a/orangecontrib/text/twitter.py b/orangecontrib/text/twitter.py
--- a/orangecontrib/text/twitter.py
+++ b/orangecontrib/text/twitter.py
@@ -71,7 +71,8 @@ class TwitterAPI:
 
     tv = data.TimeVariable('Date')
     metas = [
-        (data.StringVariable('Content'), lambda doc: doc.full_text),
+        (data.StringVariable('Content'), lambda doc: doc.full_text if not
+        doc.retweeted else doc.text),
         (tv, lambda doc: TwitterAPI.tv.parse(doc.created_at.isoformat())),
         (data.DiscreteVariable('Language'), lambda doc: doc.lang),
         (data.DiscreteVariable('Location'), lambda doc: getattr(doc.place, 'country_code', None)),
@@ -188,7 +189,8 @@ def search_authors(self, authors, *, max_tweets=0, collecting=False):
         if not isinstance(authors, list):
             authors = [authors]
 
-        cursors = [tweepy.Cursor(self.api.user_timeline, screen_name=a)
+        cursors = [tweepy.Cursor(self.api.user_timeline, screen_name=a,
+                                 tweet_mode='extended')
                    for a in authors]
         corpus, count = self.fetch(cursors, max_tweets)
         self.append_history('Author', authors, None, None, count)


Test patch:
Repository: biolab/orange3-text

Issue:
Topic Modeling: HDP fails
<!--
This is an issue template. Please fill in the relevant details in the
sections below.
-->

##### Text version
<!-- From menu _Options→Add-ons→Orange3-Text_ or code `orangecontrib.text.version.full_version` -->
0.6.0

##### Orange version
<!-- From menu _Help→About→Version_ or code `Orange.version.full_version` -->
3.21.dev

##### Expected behavior
HDP works.


##### Actual behavior
HDP crashes the widget.


##### Steps to reproduce the behavior
Corpus - Topic Modeling. Select HDP.


##### Additional info (worksheets, data, screenshots, ...)
Upon checking the code, model seems to be fit several times instead of once. The first result is empty, hence the widget fails. Yet to debug further. 




Discussion:
@robertcv You might also know how to fix this.

Resolving patch:
diff --git a/orangecontrib/text/topics/topics.py b/orangecontrib/text/topics/topics.py
--- a/orangecontrib/text/topics/topics.py
+++ b/orangecontrib/text/topics/topics.py
@@ -139,14 +139,15 @@ def get_top_words_by_id(self, topic_id, num_of_words=10):
 
     def _topics_words(self, num_of_words):
         """ Returns list of list of topic words. """
-        x = self.model.show_topics(-1, num_of_words, formatted=False)
+        x = self.model.show_topics(self.num_topics, num_of_words, formatted=False)
         # `show_topics` method return a list of `(topic_number, topic)` tuples,
         # where `topic` is a list of `(word, probability)` tuples.
         return [[i[0] for i in topic[1]] for topic in x]
 
     def _topics_weights(self, num_of_words):
         """ Returns list of list of topic weights. """
-        topics = self.model.show_topics(-1, num_of_words, formatted=False)
+        topics = self.model.show_topics(self.num_topics, num_of_words,
+                                        formatted=False)
         # `show_topics` method return a list of `(topic_number, topic)` tuples,
         # where `topic` is a list of `(word, probability)` tuples.
         return [[i[1] for i in t[1]] for t in topics]


Test patch:
diff --git a/orangecontrib/text/tests/test_topic_modeling.py b/orangecontrib/text/tests/test_topic_modeling.py
--- a/orangecontrib/text/tests/test_topic_modeling.py
+++ b/orangecontrib/text/tests/test_topic_modeling.py
@@ -23,6 +23,11 @@ def test_get_topic_table_by_id(self):
         # self.assertAlmostEqual(topic1.W.sum(), 1.)
         self.assertFalse(any(topic1.W == np.nan))
 
+    def test_get_all_topics(self):
+        self.model.fit(self.corpus)
+        topics = self.model.get_all_topics_table()
+        self.assertEqual(len(topics.domain), self.model.num_topics)
+
     def test_top_words_by_topic(self):
         self.model.fit(self.corpus)
         words, _ = self.model.get_top_words_by_id(1, num_of_words=10)
Repository: biolab/orange3-text

Issue:
PubMed importer prints warnings: Could not parse "2017 mar/apr" into a date.
```
/usr/local/lib/python3.6/site-packages/orangecontrib/text/pubmed.py:80: RuntimeWarning: Could not parse "2017 nov/dec" into a date.
  RuntimeWarning
/usr/local/lib/python3.6/site-packages/orangecontrib/text/pubmed.py:80: RuntimeWarning: Could not parse "2017 mar/apr" into a date.
  RuntimeWarning
/usr/local/lib/python3.6/site-packages/orangecontrib/text/pubmed.py:80: RuntimeWarning: Could not parse "Sep 2016" into a date.
  RuntimeWarning
/usr/local/lib/python3.6/site-packages/orangecontrib/text/pubmed.py:80: RuntimeWarning: Could not parse "2017 jan/feb" into a date.
  RuntimeWarning
```


Discussion:
Cloud you please provide a screenshot of the workflow and the query for Pubmed widget as well?
Please see the workflow, the query, these messages, and also an overall failure message that occurs:

![orange-pubmed-failure](https://user-images.githubusercontent.com/271906/41679884-88df27c4-7484-11e8-9a9d-5ccb9b0e8cbf.png)

The failure is due to shelve module that apparently has some limits on how many items can be written. https://stackoverflow.com/questions/4995162/python-shelve-dbm-error

The warning is because the date is not written in ISO. Converting other datetime formats into ISO should be done in core Orange and handled later in Text.

Resolving patch:
diff --git a/orangecontrib/text/widgets/owpubmed.py b/orangecontrib/text/widgets/owpubmed.py
--- a/orangecontrib/text/widgets/owpubmed.py
+++ b/orangecontrib/text/widgets/owpubmed.py
@@ -206,7 +206,7 @@ def __init__(self):
         label = gui.label(h_box, self, 'Retrieve')
         label.setMaximumSize(label.sizeHint())
         self.num_records_input = gui.spin(h_box, self, 'num_records',
-                                          minv=1, maxv=100000)
+                                          minv=1, maxv=10000)
         self.max_records_label = gui.label(h_box, self, 'records from /.')
         self.max_records_label.setMaximumSize(self.max_records_label
                                               .sizeHint())


Test patch:
Repository: biolab/orange3-text

Issue:
Saving a tab file after using 'import documents widget'
<!--
This is an issue template. Please fill in the relevant details in the
sections below.
-->

##### Text version
<!-- From menu _Options→Add-ons→Orange3-Text_ or code `orangecontrib.text.version.full_version` -->
0.6.0


##### Orange version
<!-- From menu _Help→About→Version_ or code `Orange.version.full_version` -->
3.20.1 on Mac. 

##### Expected behavior
"Import documents to create corpus". Send corpus to 'Save file", using *.tab format, to save the results in a folder. Using Corpus widget, can open this new tab file, with text features listed below. 


##### Actual behavior
Import is successful and I can use the corpus in workflows.
But if I try to open the new tab file as a Corpus, then I get:
-Two error messages: "Corpus doesn't have any text features"
-Error message: "Can't read file ... line contains NULL byte"



##### Steps to reproduce the behavior
I have tried:
1. opening corpus in new orange session
2. saving corpus as *.csv or .xlsx
3. trying different folders for import documents
4. opening tab file first as folder and then linking it to a Corpus widget
5. inspecting *.tab file in text editor or excel for errors.
6. trying the same versions in Windows as well as Mac


##### Additional info (worksheets, data, screenshots, ...)
The idea is to avoid having to start each session with importing documents, by starting from a tab file



Discussion:
I have tried this and it works fine for me.
Do you perhaps have some minimal dataset that you can share so we can test it?
Hi, thanks for having a look at this. 
I have cut it down to a small dataset of five documents, where the error replicates. 
One potential work-around though - if I save it as a pickle file then it will re-open it with the Corpus widget. corpus for replicating failed tab text file.zip
[corpus for replicating failed tab text file.zip](https://github.com/biolab/orange3-text/files/3112902/corpus.for.replicating.failed.tab.text.file.zip)
sorry, it didn't upload the first time
@lawrencerowland You a right. Now I can replicate this. I assume it is because of the faulty pdf (2009 Prioritisation by WSTF Weighted Shortest skimmed interesting). Still, Import Documents should probably handle this while importing. We will have a look.

Resolving patch:
diff --git a/orangecontrib/text/import_documents.py b/orangecontrib/text/import_documents.py
--- a/orangecontrib/text/import_documents.py
+++ b/orangecontrib/text/import_documents.py
@@ -138,7 +138,7 @@ def read_file(self):
             for lt_obj in layout:
                 if isinstance(lt_obj, LTTextBox) or isinstance(lt_obj, LTTextLine):
                     extracted_text.append(lt_obj.get_text())
-        self.content = ' '.join(extracted_text)
+        self.content = ' '.join(extracted_text).replace('\x00', '')
 
 
 class XmlReader(Reader):


Test patch: