Repository: piskvorky/gensim

Issue:
segment_wiki doesn't extract only plain text
The `segment_wiki.py` script doesn't filter formatting properly. Its plain text output contains rubbish like `{| class=\"wikitable\" style=\"text-align: center`, for example in the article `Academy Award for Best Production Design`, but also others.

This is contrary to the stated mission of the script.


Discussion:
This is connected with header part of table and styles, it really can be cut.
Way to found this: `grep -o -P ".{0,50}class=.{0,70}" segmentwiki_output.json`
Yeah, eyeballing search results for `class=` or even just `{` and `|` probably enough -- there's a low chance these would appear in a normal plain text.
Adding  ```RE_P16 = re.compile(r'{\|.*class=.*', re.UNICODE) ``` [here](https://github.com/RaRe-Technologies/gensim/blob/develop/gensim/corpora/wikicorpus.py#L61) will solve this problem.
@chaitaliSaini can you submit PR with fix & add tests?
Sure.
I dont think the problem can be solved just by checking for {| or class=[^=], because some other markup is also not removed, for eg,
```
uctive votes of no confidence have been attempted, and only one has been successful.\n\n{\nborder=0 cellpadding=\"1\" cellspacing=\"1\"\n\n '''Date'''\n '''Opposition candidate (party)'''\n '''Chancellor (party)
```
```
nstable: italics''odd\u2013odd isotopes coloured pink\n\n  3 \nlithium      \n 2    \n \u2014 \n \n style=\"background:pink;\"\n\u00a0\n\n 11 \nsodium       \n 1     \n \u2014 \n\n\u00a0\n\u00a0\n\n 19 \npot
```
```
ructures such as N-substituted aziridines (quaternary ammonium salts are resolvable).\n\n |76px\n | style=\"font-size:200%\" |\u00a0\u21cc\u00a0\n |76px\n |-\n | colspan=3 style=\"font-size:small\" |Inversio
```
```
hich was carried out in November 2005 on a representative 1% of the total population.\n\n\n class=\"wikitable sortable\"\n Rank\n Name\n Boys for Every 100 Girls\n\n1 \n Jiangxi \n143\n\n2 \n Henan \n142\n\n3
```
```
ingement and an out of court settlement was reached. As Page explained:\n\n\n\n |- \n | rowspan='2' style='background-color: #fff' | ''Presence''\n | \"'''Nobody's Fault but Mine'''\"\n | Page, Plant \n | \"
```
@chaitaliSaini thanks, probably we need to go through all preprocessing steps and fix the problem with html extraction.

Resolving patch:
diff --git a/gensim/corpora/wikicorpus.py b/gensim/corpora/wikicorpus.py
--- a/gensim/corpora/wikicorpus.py
+++ b/gensim/corpora/wikicorpus.py
@@ -68,9 +68,9 @@
 """Math content."""
 RE_P11 = re.compile(r'<(.*?)>', re.DOTALL | re.UNICODE)
 """All other tags."""
-RE_P12 = re.compile(r'\n(({\|)|(\|-)|(\|}))(.*?)(?=\n)', re.UNICODE)
+RE_P12 = re.compile(r'(({\|)|(\|-(?!\d))|(\|}))(.*?)(?=\n)', re.UNICODE)
 """Table formatting."""
-RE_P13 = re.compile(r'\n(\||\!)(.*?\|)*([^|]*?)', re.UNICODE)
+RE_P13 = re.compile(r'(?<=(\n[ ])|(\n\n)|([ ]{2})|(.\n)|(.\t))(\||\!)([^[\]\n]*?\|)*', re.UNICODE)
 """Table cell formatting."""
 RE_P14 = re.compile(r'\[\[Category:[^][]*\]\]', re.UNICODE)
 """Categories."""
@@ -78,7 +78,12 @@
 """Remove File and Image templates."""
 RE_P16 = re.compile(r'\[{2}(.*?)\]{2}', re.UNICODE)
 """Capture interlinks text and article linked"""
-
+RE_P17 = re.compile(
+    r'(\n.{0,4}((bgcolor)|(\d{0,1}[ ]?colspan)|(rowspan)|(style=)|(class=)|(align=)|(scope=))(.*))|'
+    '(^.{0,2}((bgcolor)|(\d{0,1}[ ]?colspan)|(rowspan)|(style=)|(class=)|(align=))(.*))',
+    re.UNICODE
+)
+"""Table markup"""
 IGNORED_NAMESPACES = [
     'Wikipedia', 'Category', 'File', 'Portal', 'Template',
     'MediaWiki', 'User', 'Help', 'Book', 'Draft', 'WikiProject',
@@ -185,10 +190,14 @@ def remove_markup(text, promote_remaining=True, simplify_links=True):
         if simplify_links:
             text = re.sub(RE_P6, '\\2', text)  # simplify links, keep description only
         # remove table markup
-
-        text = text.replace('||', '\n|')  # each table cell on a separate line
+        text = text.replace("!!", "\n|")  # each table head cell on a separate line
+        text = text.replace("|-||", "\n|")  # for cases where a cell is filled with '-'
         text = re.sub(RE_P12, '\n', text)  # remove formatting lines
-        text = re.sub(RE_P13, '\n\\3', text)  # leave only cell content
+        text = text.replace('|||', '|\n|')  # each table cell on a separate line(where |{{a|b}}||cell-content)
+        text = text.replace('||', '\n|')  # each table cell on a separate line
+        text = re.sub(RE_P13, '\n', text)  # leave only cell content
+        text = re.sub(RE_P17, '\n', text)  # remove formatting lines
+
         # remove empty mark-up
         text = text.replace('[]', '')
         # stop if nothing changed between two iterations or after a fixed number of iterations


Test patch:
diff --git a/gensim/test/test_corpora.py b/gensim/test/test_corpora.py
--- a/gensim/test/test_corpora.py
+++ b/gensim/test/test_corpora.py
@@ -645,6 +645,18 @@ def test_max_token_len_set(self):
         corpus = self.corpus_class(self.enwiki, processes=1, token_max_len=16, lemmatize=False)
         self.assertTrue(u'collectivization' in next(corpus.get_texts()))
 
+    def test_removed_table_markup(self):
+        """
+        Check if all the table markup has been removed.
+        """
+        enwiki_file = datapath('enwiki-table-markup.xml.bz2')
+        corpus = self.corpus_class(enwiki_file)
+        texts = corpus.get_texts()
+        table_markup = ["style", "class", "border", "cellspacing", "cellpadding", "colspan", "rowspan"]
+        for text in texts:
+            for word in table_markup:
+                self.assertTrue(word not in text)
+
     # #TODO: sporadic failure to be investigated
     # def test_get_texts_returns_generator_of_lists(self):
     #     corpus = self.corpus_class(self.enwiki)
diff --git a/gensim/test/test_data/enwiki-table-markup.xml.bz2 b/gensim/test/test_data/enwiki-table-markup.xml.bz2
new file mode 100644
Binary files /dev/null and b/gensim/test/test_data/enwiki-table-markup.xml.bz2 differ