Repository: piskvorky/gensim

Issue:
'NoneType' object is not iterable when using summarize
When using `gensim.summarization.summarize`, `TypeError: 'NoneType' object is not iterable` is occasionally thrown when summarizing random complaints in [this dataset of consumer complaints](https://catalog.data.gov/dataset/consumer-complaint-database). I'm not sure what it is about the strings that causes this error to be thrown but I feel the error should not be thrown regardless. Here is a self-contained example:

```
from gensim.summarization import summarize
text = "To whom it may concern, Only after weeks of using my new Wal-Mart master " \
    "card was I given the option to opt out of personal information sharing including :" \
    " Social Security #, income, account balance, payment history, credit history and credit" \
    " scores. According to the supervising operator at the wal-mart credit card call center my" \
    " personal information had already been shared. Furthermore, the letter informing me of my " \
    " rights was apart from my billing statement. Seemingly hidden between some flyers in the same" \
    " envelope. I almost threw it in the recycle bin without seeing it. If this is not illegal it " \
    "certainly appears to be an attempt to deceive me and disseminate my private information without " \
    "my permission. Thank you for your time and effort. Sincerely, XXXX XXXX"
summary = summarize(text)
```

I pasted the stack trace below but I'm sure you'll see it when running the above example.
```
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-1-15020c19bb47> in <module>()
      1 from gensim.summarization import summarize
      2 text = "To whom it may concern, Only after weeks of using my new Wal-Mart master "     "card was I given the option to opt out of personal information sharing including :"     " Social Security #, income, account balance, payment history, credit history and credit"     " scores. According to the supervising operator at the wal-mart credit card call center my"     " personal information had already been shared. Furthermore, the letter informing me of my "     " rights was apart from my billing statement. Seemingly hidden between some flyers in the same"     " envelope. I almost threw it in the recycle bin without seeing it. If this is not illegal it "     "certainly appears to be an attempt to deceive me and disseminate my private information without "    "my permission. Thank you for your time and effort. Sincerely, XXXX XXXX"
----> 3 summary = summarize(text)

/usr/local/lib/python3.6/site-packages/gensim/summarization/summarizer.py in summarize(text, ratio, word_count, split)
    213 
    214     # Extracts the most important sentences with the selected criterion.
--> 215     extracted_sentences = _extract_important_sentences(sentences, corpus, most_important_docs, word_count)
    216 
    217     # Sorts the extracted sentences by apparition order in the original text.

/usr/local/lib/python3.6/site-packages/gensim/summarization/summarizer.py in _extract_important_sentences(sentences, corpus, important_docs, word_count)
    112 
    113 def _extract_important_sentences(sentences, corpus, important_docs, word_count):
--> 114     important_sentences = _get_important_sentences(sentences, corpus, important_docs)
    115 
    116     # If no "word_count" option is provided, the number of sentences is

/usr/local/lib/python3.6/site-packages/gensim/summarization/summarizer.py in _get_important_sentences(sentences, corpus, important_docs)
     87     hashable_corpus = _build_hasheable_corpus(corpus)
     88     sentences_by_corpus = dict(zip(hashable_corpus, sentences))
---> 89     return [sentences_by_corpus[tuple(important_doc)] for important_doc in important_docs]
     90 
     91 

TypeError: 'NoneType' object is not iterable
```

I'm assuming this is a bug but there's no example in the gensim ~~documentation~~ api page so I'm not entirely sure if I'm using the function correctly.


Discussion:
Thanks for reporting.

@menshikh-iv let's link to the examples from the documentation (I think there was a blog post or notebook for summarization, at the very least).
@nabergh Docs:  [ipython tutorial](https://github.com/RaRe-Technologies/gensim/blob/develop/docs/notebooks/summarization_tutorial.ipynb) and [blogpost](https://rare-technologies.com/text-summarization-with-gensim/).

But you use it correctly, looks like a bug, @olavurmortensen please look at this problem.

@menshikh-iv I wrote that tutorial on `summarize`, but haven't had anything to do with the code. Can't help you here.
Ok, ping @fedelopez77 @fbarrios 
John Mercer sent me an email

> hey there :) this happens when there is period without space after it, e.g. text = "To whom it may conce**rn.we**eks of using my new Wal-Mart master card. was I given the option to opt out of personal information sharing including."
summary = summarize(text)

> and is fixed when adding space:

> text = "To whom it may conce**rn. we**eks of using my new Wal-Mart master card. was I given the option to opt out of personal information sharing including."
summary = summarize(text)

@nabergh please check this
I just checked this out and it seems like John is correct. If there is a period followed by no space in some cases the same error is thrown, which can be fixed by adding a space after the period. Although I do not agree with this design decision (I think a more informative error should be thrown at the very least), this case does not pertain to the example I posted above and does not solve this issue.
Looks like a bug to me, not a design decision.

Although probably related to the bug in this ticket, is my guess, and a good lead.
I''m confirming this bug which became apparent with the following text which doesn't have a dot followed by no space. 
```
Digital rights (and wrongs)
Attempting to grasp the many conflicts and proposed safeguards for intellectual
	property is extremely difficult. Legal, political, economic, and
	cultural issues-both domestic and international-loom large, almost
	dwarfing the daunting technological challenges. Solutions devised by
	courts and legislatures and regulatory agencies are always late out of
	the blocks and fall ever farther behind. Recently proposed legislation
	only illustrates the depth and complexity of the problem
```

In the meantime I'm using this function to workaround this bug
```
from gensim.summarization.textcleaner import clean_text_by_sentences
from gensim.summarization.summarizer import _build_corpus, summarize_corpus

def _summarize_if_possible(text):
    sentences = clean_text_by_sentences(text)
    if len(sentences) < 10:
        return text
    corpus = _build_corpus(sentences)
    most_important_docs = summarize_corpus(corpus, ratio=1)
    if most_important_docs is None:
        return text
    return summarize(text, word_count=600)
```

Here's another self contained example
```
from gensim.summarization import summarize

text = 'Digital rights (and wrongs)\n'
text += 'Attempting to grasp the many conflicts and proposed '
text += 'safeguards for intellectual\n\t'
text += 'property is extremely difficult. Legal, political, '
text += 'economic, and\n\tcultural issues-both domestic and '
text += 'international-loom large, almost\n\tdwarfing the '
text += 'daunting technological challenges. Solutions devised by\n'
text += '\tcourts and legislatures and regulatory agencies are '
text += 'always late out of\n\tthe blocks and fall ever farther '
text += 'behind. Recently proposed legislation\n\t'
text += 'only illustrates the depth and complexity of the problem\n'

assert len(text) == 514
summarize(text)
```
I'll look into it during the day. Hopefully I can propose a PR fixing this by today or tomorrow :) 

Resolving patch:
diff --git a/gensim/summarization/summarizer.py b/gensim/summarization/summarizer.py
--- a/gensim/summarization/summarizer.py
+++ b/gensim/summarization/summarizer.py
@@ -147,7 +147,7 @@ def summarize_corpus(corpus, ratio=0.2):
     # If the corpus is empty, the function ends.
     if len(corpus) == 0:
         logger.warning("Input corpus is empty.")
-        return
+        return []
 
     # Warns the user if there are too few documents.
     if len(corpus) < INPUT_MIN_LENGTH:
@@ -157,10 +157,11 @@ def summarize_corpus(corpus, ratio=0.2):
     _set_graph_edge_weights(graph)
     _remove_unreachable_nodes(graph)
 
-    # Cannot calculate eigenvectors if number of unique words in text < 3. Warns user to add more text. The function ends.
+    # Cannot calculate eigenvectors if number of unique documents in corpus < 3.
+    # Warns user to add more text. The function ends.
     if len(graph.nodes()) < 3:
         logger.warning("Please add more sentences to the text. The number of reachable nodes is below 3")
-        return
+        return []
 
     pagerank_scores = _pagerank(graph)
 
@@ -197,7 +198,7 @@ def summarize(text, ratio=0.2, word_count=None, split=False):
     # If no sentence could be identified, the function ends.
     if len(sentences) == 0:
         logger.warning("Input text is empty.")
-        return
+        return [] if split else u""
 
     # If only one sentence is present, the function raises an error (Avoids ZeroDivisionError).
     if len(sentences) == 1:
@@ -211,6 +212,11 @@ def summarize(text, ratio=0.2, word_count=None, split=False):
 
     most_important_docs = summarize_corpus(corpus, ratio=ratio if word_count is None else 1)
 
+    # If couldn't get important docs, the algorithm ends.
+    if not most_important_docs:
+        logger.warning("Couldn't get relevant sentences.")
+        return [] if split else u""
+
     # Extracts the most important sentences with the selected criterion.
     extracted_sentences = _extract_important_sentences(sentences, corpus, most_important_docs, word_count)
 


Test patch:
diff --git a/gensim/test/test_summarization.py b/gensim/test/test_summarization.py
--- a/gensim/test/test_summarization.py
+++ b/gensim/test/test_summarization.py
@@ -23,26 +23,24 @@
 
 class TestSummarizationTest(unittest.TestCase):
 
-    def test_text_summarization(self):
+    def _get_text_from_test_data(self, file):
         pre_path = os.path.join(os.path.dirname(__file__), 'test_data')
+        with utils.smart_open(os.path.join(pre_path, file), mode="r") as f:
+            return f.read()
 
-        with utils.smart_open(os.path.join(pre_path, "mihalcea_tarau.txt"), mode="r") as f:
-            text = f.read()
+    def test_text_summarization(self):
+        text = self._get_text_from_test_data("mihalcea_tarau.txt")
 
         # Makes a summary of the text.
         generated_summary = summarize(text)
 
         # To be compared to the method reference.
-        with utils.smart_open(os.path.join(pre_path, "mihalcea_tarau.summ.txt"), mode="r") as f:
-            summary = f.read()
+        summary = self._get_text_from_test_data("mihalcea_tarau.summ.txt")
 
         self.assertEqual(generated_summary, summary)
 
     def test_corpus_summarization(self):
-        pre_path = os.path.join(os.path.dirname(__file__), 'test_data')
-
-        with utils.smart_open(os.path.join(pre_path, "mihalcea_tarau.txt"), mode="r") as f:
-            text = f.read()
+        text = self._get_text_from_test_data("mihalcea_tarau.txt")
 
         # Generate the corpus.
         sentences = text.split("\n")
@@ -54,9 +52,8 @@ def test_corpus_summarization(self):
         selected_documents = summarize_corpus(corpus)
 
         # They are compared to the method reference.
-        with utils.smart_open(os.path.join(pre_path, "mihalcea_tarau.summ.txt"), mode="r") as f:
-            summary = f.read()
-            summary = summary.split('\n')
+        summary = self._get_text_from_test_data("mihalcea_tarau.summ.txt")
+        summary = summary.split('\n')
 
         # Each sentence in the document selection has to be in the model summary.
         for doc_number, document in enumerate(selected_documents):
@@ -67,43 +64,29 @@ def test_corpus_summarization(self):
             self.assertTrue(any(all(word in sentence for word in words)) for sentence in summary)
 
     def test_summary_from_unrelated_sentences(self):
-        # Tests that the summarization of a text with unrelated sentences does not raise an exception.
-        pre_path = os.path.join(os.path.dirname(__file__), 'test_data')
-
-        with utils.smart_open(os.path.join(pre_path, "testsummarization_unrelated.txt"), mode="r") as f:
-            text = f.read()
-
+        # Tests that the summarization of a text with unrelated sentences is not empty string.
+        text = self._get_text_from_test_data("testsummarization_unrelated.txt")
         generated_summary = summarize(text)
+        self.assertNotEqual(generated_summary, u"")
 
-        self.assertNotEqual(generated_summary, None)
-
-    def test_text_summarization_raises_exception_on_short_input_text(self):
-        pre_path = os.path.join(os.path.dirname(__file__), 'test_data')
-
-        with utils.smart_open(os.path.join(pre_path, "testsummarization_unrelated.txt"), mode="r") as f:
-            text = f.read()
+    def test_text_summarization_on_short_input_text_is_empty_string(self):
+        text = self._get_text_from_test_data("testsummarization_unrelated.txt")
 
         # Keeps the first 8 sentences to make the text shorter.
         text = "\n".join(text.split('\n')[:8])
 
-        self.assertTrue(summarize(text) is not None)
-
-    def test_text_summarization_returns_input_on_single_input_sentence(self):
-        pre_path = os.path.join(os.path.dirname(__file__), 'test_data')
+        self.assertNotEqual(summarize(text), u"")
 
-        with utils.smart_open(os.path.join(pre_path, "testsummarization_unrelated.txt"), mode="r") as f:
-            text = f.read()
+    def test_text_summarization_raises_exception_on_single_input_sentence(self):
+        text = self._get_text_from_test_data("testsummarization_unrelated.txt")
 
         # Keeps the first sentence only.
         text = text.split('\n')[0]
 
         self.assertRaises(ValueError, summarize, text)
 
-    def test_corpus_summarization_raises_exception_on_short_input_text(self):
-        pre_path = os.path.join(os.path.dirname(__file__), 'test_data')
-
-        with utils.smart_open(os.path.join(pre_path, "testsummarization_unrelated.txt"), mode="r") as f:
-            text = f.read()
+    def test_corpus_summarization_is_not_empty_list_on_short_input_text(self):
+        text = self._get_text_from_test_data("testsummarization_unrelated.txt")
 
         # Keeps the first 8 sentences to make the text shorter.
         sentences = text.split('\n')[:8]
@@ -113,19 +96,19 @@ def test_corpus_summarization_raises_exception_on_short_input_text(self):
         dictionary = Dictionary(tokens)
         corpus = [dictionary.doc2bow(sentence_tokens) for sentence_tokens in tokens]
 
-        self.assertTrue(summarize_corpus(corpus) is not None)
+        self.assertNotEqual(summarize_corpus(corpus), [])
 
-    def test_empty_text_summarization_none(self):
-        self.assertTrue(summarize("") is None)
+    def test_empty_text_summarization_is_empty_string(self):
+        self.assertEquals(summarize(""), u"")
 
-    def test_empty_corpus_summarization_is_none(self):
-        self.assertTrue(summarize_corpus([]) is None)
+    def test_empty_text_summarization_with_split_is_empty_list(self):
+        self.assertEquals(summarize("", split=True), [])
 
-    def test_corpus_summarization_ratio(self):
-        pre_path = os.path.join(os.path.dirname(__file__), 'test_data')
+    def test_empty_corpus_summarization_is_empty_list(self):
+        self.assertEquals(summarize_corpus([]), [])
 
-        with utils.smart_open(os.path.join(pre_path, "mihalcea_tarau.txt"), mode="r") as f:
-            text = f.read()
+    def test_corpus_summarization_ratio(self):
+        text = self._get_text_from_test_data("mihalcea_tarau.txt")
 
         # Generate the corpus.
         sentences = text.split('\n')
@@ -142,10 +125,7 @@ def test_corpus_summarization_ratio(self):
             self.assertEqual(len(selected_docs), expected_summary_length)
 
     def test_repeated_keywords(self):
-        pre_path = os.path.join(os.path.dirname(__file__), 'test_data')
-
-        with utils.smart_open(os.path.join(pre_path, "testrepeatedkeywords.txt")) as f:
-            text = f.read()
+        text = self._get_text_from_test_data("testrepeatedkeywords.txt")
 
         kwds = keywords(text)
         self.assertTrue(len(kwds.splitlines()))
@@ -157,10 +137,7 @@ def test_repeated_keywords(self):
         self.assertTrue(len(kwds_lst))
 
     def test_keywords_runs(self):
-        pre_path = os.path.join(os.path.dirname(__file__), 'test_data')
-
-        with utils.smart_open(os.path.join(pre_path, "mihalcea_tarau.txt")) as f:
-            text = f.read()
+        text = self._get_text_from_test_data("mihalcea_tarau.txt")
 
         kwds = keywords(text)
         self.assertTrue(len(kwds.splitlines()))
@@ -171,11 +148,8 @@ def test_keywords_runs(self):
         kwds_lst = keywords(text, split=True)
         self.assertTrue(len(kwds_lst))
 
-    def test_low_distinct_words_corpus_summarization_is_none(self):
-        pre_path = os.path.join(os.path.dirname(__file__), 'test_data')
-
-        with utils.smart_open(os.path.join(pre_path, "testlowdistinctwords.txt"), mode="r") as f:
-            text = f.read()
+    def test_low_distinct_words_corpus_summarization_is_empty_list(self):
+        text = self._get_text_from_test_data("testlowdistinctwords.txt")
 
         # Generate the corpus.
         sentences = text.split("\n")
@@ -183,8 +157,15 @@ def test_low_distinct_words_corpus_summarization_is_none(self):
         dictionary = Dictionary(tokens)
         corpus = [dictionary.doc2bow(sentence_tokens) for sentence_tokens in tokens]
 
-        self.assertTrue(summarize_corpus(corpus) is None)
+        self.assertEquals(summarize_corpus(corpus), [])
+
+    def test_low_distinct_words_summarization_is_empty_string(self):
+        text = self._get_text_from_test_data("testlowdistinctwords.txt")
+        self.assertEquals(summarize(text), u"")
 
+    def test_low_distinct_words_summarization_with_split_is_empty_list(self):
+        text = self._get_text_from_test_data("testlowdistinctwords.txt")
+        self.assertEquals(summarize(text, split=True), [])
 
 if __name__ == '__main__':
     logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.DEBUG)