piskvorky/gensim — pull request #83 by aweinstein (2012-03-26 18:01:55+00:00)
Title: Improve wikimedia markup filtering.

I am improving the filtering of:

1. Templates
The current implementation uses a regular expression. This is problematic when there are nested templates. I am replacing this by the function `remove_template`. This function find the location of the opening '{{' and closing '}}' for each template. Then it uses these locations to remove the templates from the text.

2. 'File:'/'Image:' markup
The current regular expression used to filter these markups sometimes fails. I am replacing this by the function `remove_file`. This function first find all the 'File:'/'Image:' markups using a regular expression, and then replace each instance by their corresponding captions.