Repository: piskvorky/gensim

Issue:
KeyError: Popping a non-existing value from `_keywords`
OS X 10.11.1
Python 2.7.10
gensim 0.12.4

To reproduce this in Python shell:

```
>>> import gensim.summarization
>>> t = "Victor S. Sage Compare Sage 50c Editions Find accounting software that's right for your business Every product comes with anytime, anywhere online access; automatic updates; access to unlimited support; access to built-in credit card processing and payroll; and advanced reporting. Three solutions for your business 1 user From $249/year Buy now Free Trial 1-5 users From $299/year Buy now Free Trial 3-40 users From $1,199/year Buy now Free Trial Essential Accounting Accounts payable, accounts receivable, cash management check check check open check check check check check check check check check check check check check check check check check check check check check check check check check check check check check check check check check check check check check check check check check check check check check check check check Advanced Accounting Automated tasks, audit trail, budgeting, change order processing check check open check check check check check check check check check check check check check check check check check check check check check check check check check check check check In-depth Accounting Fast processing, industry-specific features, workflow management check open check check check check check check check check check check check Disclaimers open * This product is backed by a no-risk guarantee for first-time Sage 50 customers. If, within 60 days of purchase, you are not convinced that Sage 50 is the best accounting program for your business, we will refund your money (less and rebate you have received for this purchase). Dated proof of purchase and return of product is required. For details, call 877-481-0341."
>>> import gensim.summarization
>>> keywords = gensim.summarization.keywords(t, pos_filter=[], ratio=0.2, lemmatize=True, scores=True)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/site-packages/gensim/summarization/keywords.py", line 229, in keywords
    combined_keywords = _get_combined_keywords(keywords, text.split())
  File "/usr/local/lib/python2.7/site-packages/gensim/summarization/keywords.py", line 171, in _get_combined_keywords
    _keywords.pop(keyword)
KeyError: u'check'
```



Discussion:

Resolving patch:
diff --git a/gensim/summarization/keywords.py b/gensim/summarization/keywords.py
--- a/gensim/summarization/keywords.py
+++ b/gensim/summarization/keywords.py
@@ -164,7 +164,7 @@ def _get_combined_keywords(_keywords, split_text):
                 result.append(word)   # appends last word if keyword and doesn't iterate
             for j in xrange(i + 1, len_text):
                 other_word = _strip_word(split_text[j])
-                if other_word in _keywords and other_word == split_text[j]:
+                if other_word in _keywords and other_word == split_text[j] and not other_word in combined_word:
                     combined_word.append(other_word)
                 else:
                     for keyword in combined_word:


Test patch:
diff --git a/gensim/test/test_data/testrepeatedkeywords.txt b/gensim/test/test_data/testrepeatedkeywords.txt
new file mode 100644
--- /dev/null
+++ b/gensim/test/test_data/testrepeatedkeywords.txt
@@ -0,0 +1 @@
+Victor S. Sage Compare Sage 50c Editions Find accounting software that's right for your business Every product comes with anytime, anywhere online access; automatic updates; access to unlimited support; access to built-in credit card processing and payroll; and advanced reporting. Three solutions for your business 1 user From $249/year Buy now Free Trial 1-5 users From $299/year Buy now Free Trial 3-40 users From $1,199/year Buy now Free Trial Essential Accounting Accounts payable, accounts receivable, cash management check check check open check check check check check check check check check check check check check check check check check check check check check check check check check check check check check check check check check check check check check check check check check check check check check check check check Advanced Accounting Automated tasks, audit trail, budgeting, change order processing check check open check check check check check check check check check check check check check check check check check check check check check check check check check check check check In-depth Accounting Fast processing, industry-specific features, workflow management check open check check check check check check check check check check check Disclaimers open * This product is backed by a no-risk guarantee for first-time Sage 50 customers. If, within 60 days of purchase, you are not convinced that Sage 50 is the best accounting program for your business, we will refund your money (less and rebate you have received for this purchase). Dated proof of purchase and return of product is required. For details, call 877-481-0341.
diff --git a/gensim/test/test_summarization.py b/gensim/test/test_summarization.py
--- a/gensim/test/test_summarization.py
+++ b/gensim/test/test_summarization.py
@@ -130,6 +130,21 @@ def test_corpus_summarization_ratio(self):
 
             self.assertEqual(len(selected_docs), expected_summary_length)
 
+    def test_repeated_keywords(self):
+        pre_path = os.path.join(os.path.dirname(__file__), 'test_data')
+
+        with utils.smart_open(os.path.join(pre_path, "testrepeatedkeywords.txt")) as f:
+            text = f.read()
+
+        kwds = keywords(text)
+        self.assertTrue(len(kwds.splitlines()))
+
+        kwds_u = keywords(utils.to_unicode(text))
+        self.assertTrue(len(kwds_u.splitlines()))
+
+        kwds_lst = keywords(text, split=True)
+        self.assertTrue(len(kwds_lst))
+
     def test_keywords_runs(self):
         pre_path = os.path.join(os.path.dirname(__file__), 'test_data')