Repository: piskvorky/gensim

Issue:
Add `TextDirectoryCorpus` that yields one doc per file recursively read from directory
#### Description
Plain text corpuses are sometimes represented using a directory structure with, for instance a top-level directory and subdirectories that represent categories. Then within each subdirectory, each file might be a document. The nesting may run deeper in such directory structures to reflect additional categorization. The popular 20 newsgroups dataset has such a structure. Here is a subset of that structure:
```
20news-18828/
|-- alt.atheism
|   |-- 49960
|   |-- 51060
|   |-- 51119
|-- comp.graphics
|   |-- 37261
|   |-- 37913
|   |-- 37914
|   |-- 37915
|   |-- 37916
|   |-- 37917
|   |-- 37918
|-- comp.os.ms-windows.misc
|   |-- 10000
|   |-- 10001
|   |-- 10002
|   |-- 10003
|   |-- 10004
|   |-- 10005 
```

It would be useful to have a gensim corpus available to handle this sort of corpus structure.

#### Steps/Code/Corpus to Reproduce

I'm envisioning something like this:
```
class TextDirectoryCorpus(TextCorpus):
    """Read documents recursively from a directory,
    where each file is interpreted as a plain text document.
    """

    def __init__(self, input, metadata=False, min_depth=0, max_depth=None, pattern=None,
                 exclude_pattern=None, **kwargs):
        self.min_depth = min_depth
        self.max_depth = sys.maxsize if max_depth is None else max_depth
        self.pattern = pattern
        self.exclude_pattern = exclude_pattern
        super(TextDirectoryCorpus, self).__init__(input, metadata, **kwargs)
    ....

corpus = textcorpus.TextDirectoryCorpus('20_newsgroups', exclude_pattern="README.*")
paths = corpus.iter_filepaths()
texts = corpus.get_texts()
```

#### Expected Results

#### Actual Results

#### Versions
This should be available in all versions.




Discussion:
I'm not sure this is something we want in core gensim... there's no end to number of similar utility functions and their combinations we could potentially support, a Pandora's box.

I summarized my thoughts on this topic in a blog post: [API Bondage](https://rare-technologies.com/data-streaming-in-python-generators-iterators-iterables/).

@menshikh-iv @gojomo @cscorley @tmylk your thoughts?

I think It's a useful feature. I have written similar wrappers many times.
Moreover, it seems this is one of the most popular use-cases.

I understand your position @piskvorky, but where is the line after which we no longer want to add new features?
I can certainly understand not wanting to bloat a code base. I've also written similar wrappers many times. I figured since there's already a `TextCorpus`, this is a similar thing and likely just as useful. However, I can also conceive of a separate repo that aggregates together corpus classes for parsing various data sources in a variety of formats. I would think that if this shouldn't be in gensim, then the wikicorpus probably should live somewhere else as well. @piskvorky have you considered splitting out the corpora classes into another repo?

One additional benefit of grouping this and similar corpus classes: it would open up opportunities to provide the distribution used by the `wikicorpus` for arbitrary text corpora. With a bit of redesign, the chunking and distribution code could be moved to `TextCorpus` and used across all subclasses. The subclasses would then provide hooks for preprocessing (e.g. `preprocess_text` in the PR) and possibly filtering out garbage texts (e.g. those that are empty or marked as garbage during preprocessing). Or some alternative method could be provided using callbacks on the stream, or something of that sort. Doing this nicely might require using dill (#558) or using one of the more [lightweight solutions based on `copy_reg`](https://stackoverflow.com/questions/5429584/handling-the-classmethod-pickling-issue-with-copy-reg).
Yeah, a separate dedicated subpackage (or even a repo) that focuses on various efficient, streamed (parallelized?) readers, for sundry data formats, sounds good to me.

I don't mind including this "subdirs reader" as a blueprint example, it's a common use case as you say. But I'd be -1 on adding many such readers in an ad-hoc manner, at ad-hoc locations, throughout gensim. Having a clear structure and plan behind it sounds better.



Having a dedicated subpackage opens up some interesting options. What do you think of the following proposal? It's a tad bit long, but I hope it's closer to a clear structure and plan.

(1) Add a new `datasets` subpackage that will provide various `Dataset` classes. The `Dataset` class will be a simple extension of the corpus interface that includes a URL (or list of URLs) where a dataset can be obtained from, along with code for downloading them. This will satisfy #717 and provide an obvious place to put one-off or narrowly applicable corpora code. In this case, my motivation for implementing the `TextDirectoryCorpus` is really to use it for the 20 newsgroups dataset (which is a nice starter dataset for exploration and clustering use cases). I've implemented that [in this gist](https://gist.github.com/macks22/3754f132efc6bf6e54311c551ce675a8). That code would fit well in the proposed `datasets` subpackage.

(2) Combine the various text-based corpus classes into `gensim.corpora.textcorpus` or transfer them to the new `datasets` subpackage. From what I can tell, there are currently 8 of these, including the `WikiCorpus`. It may be best to keep the `WikiCorpus` where it is, so let's consider the other 7 for now (8 if you include `TextDirectoryCorpus`). These are:

1. `gensim.textcorpus.TextCorpus`
2. `gensim.test_miislita.CorpusMiislita`: simple case of `TextCorpus` that does stopword filtering in addition to unicode conversion, lowercasing, and tokenization. Augmenting `TextCorpus` with an optional stopwords list would allow it to subsume this. This could be included in a `Dataset` in the `datasets` package if it corresponds to some known URLs for download.
3. `gensim.word2vec.BrownCorpus`: simple case of `TextDirectoryCorpus` that does unicode conversion, lowercasing, tokenization, and accounts for the POS tags in the Brown corpus. Good candidate to move to `datasets` package.
4. `gensim.word2vec.LineSentence`: special case of `TextCorpus` in which each line, instead of being a single document, may be one or more. Also introduces an option to limit the number of lines considered. Performs unicode conversion and tokenization. This could either be reimplemented by composing or subclassing `TextCorpus` and overriding `getstream` (new variant from the PR).
5. `gensim.word2vec.Text8Corpus`: simple case of `LineSentence` with one line. Performs unicode conversion, tokenization, and whitespace stripping. Could be implemented by overriding the `getstream` method in the modified `LineSentence` discussed above.
6. `gensim.doc2vec.TaggedBrownCorpus`: almost the exact same code as `BrownCorpus` but wraps the words output in a `TaggedDocument`. Could also be moved to `datasets` package.
7. `gensim.doc2vec.TaggedLineDocument`: input is same as that to `TextCorpus`, but only does unicode conversion and tokenization before wrapping in a `TaggedDocument` with the tag derived from the line number. Since this format is specific to the `Doc2Vec` model, it makes sense to leave it where it is, but it could leverage all of the other data formats if it assumed its input was an iterator of lists of tokens.

So, common functionality includes taking in one or more text files as either file handles or FS paths, opening and reading them, performing unicode conversion and tokenization, then yielding the words in some format. Each line may be converted into one or more documents, and lowercasing and stopword removal may be performed. Additional preprocessing may occur (as in the POS tag handling). Finally, the words may be wrapped up in some other wrapper (as in `TaggedDocument`).

So to reorganize, my initial thought would be this:

`gensim.corpora.textcorpus`
-   `TextCorpus`
-   `TextDirectoryCorpus`
-   `LineSentence`
-   `Text8Corpus`

`gensim.datasets`
-    `BrownCorpus`
-    `MiislitaCorpus`
-    `TaggedBrownCorpus`

An alternative might be to put tagged corpora in a `datasets.tagged` subpackage or something like that. One could also argue that the `Text8Corpus` should go in `datasets`, but I think that format is not uncommon for word2vec datasets. So the parsing code should probably be in `textcorpus` with a simple wrapper `Dataset` class.
Awesome! Thanks for the thoughtful investigation @macks22 .

Since this looks like an architectural change, let's also include the bundling of datasets and models into the discussion. We've been wanting to include some common datasets and trained models for playing around into gensim (beyond the tiny data we have as a part of unit tests now).

Since it's related to how we handle the corpus inputs, so I think it should be discussed at the same place.
So you want to discuss how to include pre-trained models or make them accessible for download? For the datasets, I would think you could just provide code to download them from public URLs (as scikit does). For models, I'm not sure what the best approach is. It might be worth checking out how SpaCy handles that, since they do have some mechanism for [loading pre-trained GloVe word vectors](https://spacy.io/docs/usage/word-vectors-similarities).

Are these the sorts of concerns you are wanting to include in the discussion?

Resolving patch:
diff --git a/gensim/corpora/textcorpus.py b/gensim/corpora/textcorpus.py
--- a/gensim/corpora/textcorpus.py
+++ b/gensim/corpora/textcorpus.py
@@ -30,18 +30,41 @@
 from __future__ import with_statement
 
 import logging
+import os
 import random
+import re
+import sys
 
 from gensim import interfaces, utils
-from six import string_types
 from gensim.corpora.dictionary import Dictionary
+from gensim.parsing.preprocessing import STOPWORDS, RE_WHITESPACE
+from gensim.utils import deaccent, simple_tokenize
 
-logger = logging.getLogger('gensim.corpora.textcorpus')
+logger = logging.getLogger(__name__)
+
+
+def remove_stopwords(tokens, stopwords=STOPWORDS):
+    """Remove stopwords using list from `gensim.parsing.preprocessing.STOPWORDS."""
+    return [token for token in tokens if token not in stopwords]
+
+
+def remove_short(tokens, minsize=3):
+    """Remove tokens smaller than `minsize` chars, which is 3 by default."""
+    return [token for token in tokens if len(token) >= minsize]
+
+
+def lower_to_unicode(text, encoding='utf8', errors='strict'):
+    """Lowercase `text` and convert to unicode."""
+    return utils.to_unicode(text.lower(), encoding, errors)
+
+
+def strip_multiple_whitespaces(s):
+    """Collapse multiple whitespace characters into a single space."""
+    return RE_WHITESPACE.sub(" ", s)
 
 
 class TextCorpus(interfaces.CorpusABC):
-    """
-    Helper class to simplify the pipeline of getting bag-of-words vectors (= a
+    """Helper class to simplify the pipeline of getting bag-of-words vectors (= a
     gensim corpus) from plain text.
 
     This is an abstract base class: override the `get_texts()` and `__len__()`
@@ -49,58 +72,190 @@ class TextCorpus(interfaces.CorpusABC):
 
     Given a filename (or a file-like object) in constructor, the corpus object
     will be automatically initialized with a dictionary in `self.dictionary` and
-    will support the `iter` corpus method. You must only provide a correct `get_texts`
-    implementation.
+    will support the `iter` corpus method. You have a few different ways of utilizing
+    this class via subclassing or by construction with different preprocessing arguments.
+
+    The `iter` method converts the lists of tokens produced by `get_texts` to BoW format
+    using `Dictionary.doc2bow`. `get_texts` does the following:
+
+    1.  Calls `getstream` to get a generator over the texts. It yields each document in
+        turn from the underlying text file or files.
+    2.  For each document from the stream, calls `preprocess_text` to produce a list of
+        tokens; if metadata is enabled, it yields a 2-`tuple` with the document number as
+        the second element.
+
+
+    Preprocessing consists of 0+ `character_filters`, a `tokenizer`, and 0+ `token_filters`.
+
+    The preprocessing consists of calling each filter in `character_filters` with the document
+    text; unicode is not guaranteed, and if desired, the first filter should convert to unicode.
+    The output of each character filter should be another string. The output from the final
+    filter is fed to the `tokenizer`, which should split the string into a list of tokens (strings).
+    Afterwards, the list of tokens is fed through each filter in `token_filters`. The final
+    output returned from `preprocess_text` is the output from the final token filter.
+
+    So to use this class, you can either pass in different preprocessing functions using the
+    `character_filters`, `tokenizer`, and `token_filters` arguments, or you can subclass it.
+    If subclassing: override `getstream` to take text from different input sources in different
+    formats. Overrride `preprocess_text` if you must provide different initial preprocessing,
+    then call the `TextCorpus.preprocess_text` method to apply the normal preprocessing. You
+    can also overrride `get_texts` in order to tag the documents (token lists) with different
+    metadata.
+
+    The default preprocessing consists of:
+
+    1.  lowercase and convert to unicode; assumes utf8 encoding
+    2.  deaccent (asciifolding)
+    3.  collapse multiple whitespaces into a single one
+    4.  tokenize by splitting on whitespace
+    5.  remove words less than 3 characters long
+    6.  remove stopwords; see `gensim.parsing.preprocessing` for the list of stopwords
 
     """
-    def __init__(self, input=None):
-        super(TextCorpus, self).__init__()
+    def __init__(self, input=None, dictionary=None, metadata=False, character_filters=None,
+                 tokenizer=None, token_filters=None):
+        """
+        Args:
+            input (str): path to top-level directory to traverse for corpus documents.
+            dictionary (Dictionary): if a dictionary is provided, it will not be updated
+                with the given corpus on initialization. If none is provided, a new dictionary
+                will be built for the given corpus. If no corpus is given, the dictionary will
+                remain uninitialized.
+            metadata (bool): True to yield metadata with each document, else False (default).
+            character_filters (iterable of callable): each will be applied to the text of each
+                document in order, and should return a single string with the modified text.
+                For Python 2, the original text will not be unicode, so it may be useful to
+                convert to unicode as the first character filter. The default character filters
+                lowercase, convert to unicode (strict utf8), perform ASCII-folding, then collapse
+                multiple whitespaces.
+            tokenizer (callable): takes as input the document text, preprocessed by all filters
+                in `character_filters`; should return an iterable of tokens (strings).
+            token_filters (iterable of callable): each will be applied to the iterable of tokens
+                in order, and should return another iterable of tokens. These filters can add,
+                remove, or replace tokens, or do nothing at all. The default token filters
+                remove tokens less than 3 characters long and remove stopwords using the list
+                in `gensim.parsing.preprocessing.STOPWORDS`.
+        """
         self.input = input
-        self.dictionary = Dictionary()
-        self.metadata = False
-        if input is not None:
-            self.dictionary.add_documents(self.get_texts())
+        self.metadata = metadata
+
+        self.character_filters = character_filters
+        if self.character_filters is None:
+            self.character_filters = [lower_to_unicode, deaccent, strip_multiple_whitespaces]
+
+        self.tokenizer = tokenizer
+        if self.tokenizer is None:
+            self.tokenizer = simple_tokenize
+
+        self.token_filters = token_filters
+        if self.token_filters is None:
+            self.token_filters = [remove_short, remove_stopwords]
+
+        self.length = None
+        self.dictionary = None
+        self.init_dictionary(dictionary)
+
+    def init_dictionary(self, dictionary):
+        """If `dictionary` is None, initialize to an empty Dictionary, and then if there
+        is an `input` for the corpus, add all documents from that `input`. If the
+        `dictionary` is already initialized, simply set it as the corpus's `dictionary`.
+        """
+        self.dictionary = dictionary if dictionary is not None else Dictionary()
+        if self.input is not None:
+            if dictionary is None:
+                logger.info("Initializing dictionary")
+                metadata_setting = self.metadata
+                self.metadata = False
+                self.dictionary.add_documents(self.get_texts())
+                self.metadata = metadata_setting
+            else:
+                logger.info("Input stream provided but dictionary already initialized")
         else:
-            logger.warning("No input document stream provided; assuming "
-                           "dictionary will be initialized some other way.")
+            logger.warning(
+                "No input document stream provided; assuming "
+                "dictionary will be initialized some other way.")
 
     def __iter__(self):
-        """
-        The function that defines a corpus.
+        """The function that defines a corpus.
 
         Iterating over the corpus must yield sparse vectors, one for each document.
         """
-        for text in self.get_texts():
-            if self.metadata:
-                yield self.dictionary.doc2bow(text[0], allow_update=False), text[1]
-            else:
+        if self.metadata:
+            for text, metadata in self.get_texts():
+                yield self.dictionary.doc2bow(text, allow_update=False), metadata
+        else:
+            for text in self.get_texts():
                 yield self.dictionary.doc2bow(text, allow_update=False)
 
     def getstream(self):
-        return utils.file_or_filename(self.input)
+        """Yield documents from the underlying plain text collection (of one or more files).
+        Each item yielded from this method will be considered a document by subsequent
+        preprocessing methods.
+        """
+        num_texts = 0
+        with utils.file_or_filename(self.input) as f:
+            for line in f:
+                yield line
+                num_texts += 1
 
-    def get_texts(self):
+        self.length = num_texts
+
+    def preprocess_text(self, text):
+        """Apply preprocessing to a single text document. This should perform tokenization
+        in addition to any other desired preprocessing steps.
+
+        Args:
+            text (str): document text read from plain-text file.
+
+        Returns:
+            iterable of str: tokens produced from `text` as a result of preprocessing.
         """
-        Iterate over the collection, yielding one document at a time. A document
+        for character_filter in self.character_filters:
+            text = character_filter(text)
+
+        tokens = self.tokenizer(text)
+        for token_filter in self.token_filters:
+            tokens = token_filter(tokens)
+
+        return tokens
+
+    def step_through_preprocess(self, text):
+        """Yield tuples of functions and their output for each stage of preprocessing.
+        This is useful for debugging issues with the corpus preprocessing pipeline.
+        """
+        for character_filter in self.character_filters:
+            text = character_filter(text)
+            yield (character_filter, text)
+
+        tokens = self.tokenizer(text)
+        yield (self.tokenizer, tokens)
+
+        for token_filter in self.token_filters:
+            tokens = token_filter(tokens)
+            yield (token_filter, tokens)
+
+    def get_texts(self):
+        """Iterate over the collection, yielding one document at a time. A document
         is a sequence of words (strings) that can be fed into `Dictionary.doc2bow`.
+        Each document will be fed through `preprocess_text`. That method should be
+        overridden to provide different preprocessing steps. This method will need
+        to be overridden if the metadata you'd like to yield differs from the line
+        number.
 
-        Override this function to match your input (parse input files, do any
-        text preprocessing, lowercasing, tokenizing etc.). There will be no further
-        preprocessing of the words coming out of this function.
+        Returns:
+            generator of lists of tokens (strings); each list corresponds to a preprocessed
+            document from the corpus `input`.
         """
-        # Instead of raising NotImplementedError, let's provide a sample implementation:
-        # assume documents are lines in a single file (one document per line).
-        # Yield each document as a list of lowercase tokens, via `utils.tokenize`.
-        with self.getstream() as lines:
+        lines = self.getstream()
+        if self.metadata:
             for lineno, line in enumerate(lines):
-                if self.metadata:
-                    yield utils.tokenize(line, lowercase=True), (lineno,)
-                else:
-                    yield utils.tokenize(line, lowercase=True)
+                yield self.preprocess_text(line), (lineno,)
+        else:
+            for line in lines:
+                yield self.preprocess_text(line)
 
     def sample_texts(self, n, seed=None, length=None):
-        """
-        Yield n random documents from the corpus without replacement.
+        """Yield n random documents from the corpus without replacement.
 
         Given the number of remaining documents in a corpus, we need to choose n elements.
         The probability for the current element to be chosen is n/remaining.
@@ -120,12 +275,7 @@ def sample_texts(self, n, seed=None, length=None):
         Raises:
             ValueError: when n is invalid or length was set incorrectly.
         """
-        random_generator = None
-        if seed is None:
-            random_generator = random
-        else:
-            random_generator = random.Random(seed)
-
+        random_generator = random if seed is None else random.Random(seed)
         if length is None:
             length = len(self)
 
@@ -134,14 +284,18 @@ def sample_texts(self, n, seed=None, length=None):
         if not 0 <= n:
             raise ValueError("Negative sample size.")
 
-        for i, sample in enumerate(self.get_texts()):
+        for i, sample in enumerate(self.getstream()):
             if i == length:
                 break
+
             remaining_in_corpus = length - i
             chance = random_generator.randint(1, remaining_in_corpus)
             if chance <= n:
                 n -= 1
-                yield sample
+                if self.metadata:
+                    yield self.preprocess_text(sample[0]), sample[1]
+                else:
+                    yield self.preprocess_text(sample)
 
         if n != 0:
             # This means that length was set to be greater than number of items in corpus
@@ -149,9 +303,172 @@ def sample_texts(self, n, seed=None, length=None):
             raise ValueError("length greater than number of documents in corpus")
 
     def __len__(self):
-        if not hasattr(self, 'length'):
+        if self.length is None:
             # cache the corpus length
-            self.length = sum(1 for _ in self.get_texts())
+            self.length = sum(1 for _ in self.getstream())
         return self.length
-
 # endclass TextCorpus
+
+
+class TextDirectoryCorpus(TextCorpus):
+    """Read documents recursively from a directory,
+    where each file (or line of each file) is interpreted as a plain text document.
+    """
+
+    def __init__(self, input, dictionary=None, metadata=False, min_depth=0, max_depth=None,
+                 pattern=None, exclude_pattern=None, lines_are_documents=False, **kwargs):
+        """
+        Args:
+            min_depth (int): minimum depth in directory tree at which to begin searching for
+                files. The default is 0, which means files starting in the top-level directory
+                `input` will be considered.
+            max_depth (int): max depth in directory tree at which files will no longer be
+                considered. The default is None, which means recurse through all subdirectories.
+            pattern (str or Pattern): regex to use for file name inclusion; all those files *not*
+                matching this pattern will be ignored.
+            exclude_pattern (str or Pattern): regex to use for file name exclusion; all files
+                matching this pattern will be ignored.
+            lines_are_documents (bool): if True, each line of each file is considered to be a
+                document. If False (default), each file is considered to be a document.
+            kwargs: keyword arguments passed through to the `TextCorpus` constructor. This is
+                in addition to the non-kwargs `input`, `dictionary`, and `metadata`. See
+                `TextCorpus.__init__` docstring for more details on these.
+        """
+        self._min_depth = min_depth
+        self._max_depth = sys.maxsize if max_depth is None else max_depth
+        self.pattern = pattern
+        self.exclude_pattern = exclude_pattern
+        self.lines_are_documents = lines_are_documents
+        super(TextDirectoryCorpus, self).__init__(input, dictionary, metadata, **kwargs)
+
+    @property
+    def lines_are_documents(self):
+        return self._lines_are_documents
+
+    @lines_are_documents.setter
+    def lines_are_documents(self, lines_are_documents):
+        self._lines_are_documents = lines_are_documents
+        self.length = None
+
+    @property
+    def pattern(self):
+        return self._pattern
+
+    @pattern.setter
+    def pattern(self, pattern):
+        self._pattern = None if pattern is None else re.compile(pattern)
+        self.length = None
+
+    @property
+    def exclude_pattern(self):
+        return self._exclude_pattern
+
+    @exclude_pattern.setter
+    def exclude_pattern(self, pattern):
+        self._exclude_pattern = None if pattern is None else re.compile(pattern)
+        self.length = None
+
+    @property
+    def min_depth(self):
+        return self._min_depth
+
+    @min_depth.setter
+    def min_depth(self, min_depth):
+        self._min_depth = min_depth
+        self.length = None
+
+    @property
+    def max_depth(self):
+        return self._max_depth
+
+    @max_depth.setter
+    def max_depth(self, max_depth):
+        self._max_depth = max_depth
+        self.length = None
+
+    def iter_filepaths(self):
+        """Lazily yield paths to each file in the directory structure within the specified
+        range of depths. If a filename pattern to match was given, further filter to only
+        those filenames that match.
+        """
+        for depth, dirpath, dirnames, filenames in walk(self.input):
+            if self.min_depth <= depth <= self.max_depth:
+                if self.pattern is not None:
+                    filenames = (n for n in filenames if self.pattern.match(n) is not None)
+                if self.exclude_pattern is not None:
+                    filenames = (n for n in filenames if self.exclude_pattern.match(n) is None)
+
+                for name in filenames:
+                    yield os.path.join(dirpath, name)
+
+    def getstream(self):
+        """Yield documents from the underlying plain text collection (of one or more files).
+        Each item yielded from this method will be considered a document by subsequent
+        preprocessing methods.
+
+        If `lines_are_documents` was set to True, items will be lines from files. Otherwise
+        there will be one item per file, containing the entire contents of the file.
+        """
+        num_texts = 0
+        for path in self.iter_filepaths():
+            with open(path, 'rt') as f:
+                if self.lines_are_documents:
+                    for line in f:
+                        yield line.strip()
+                        num_texts += 1
+                else:
+                    yield f.read().strip()
+                    num_texts += 1
+
+        self.length = num_texts
+
+    def __len__(self):
+        if self.length is None:
+            self._cache_corpus_length()
+        return self.length
+
+    def _cache_corpus_length(self):
+        if not self.lines_are_documents:
+            self.length = sum(1 for _ in self.iter_filepaths())
+        else:
+            self.length = sum(1 for _ in self.getstream())
+# endclass TextDirectoryCorpus
+
+
+def walk(top, topdown=True, onerror=None, followlinks=False, depth=0):
+    """This is a mostly copied version of `os.walk` from the Python 2 source code.
+    The only difference is that it returns the depth in the directory tree structure
+    at which each yield is taking place.
+    """
+    islink, join, isdir = os.path.islink, os.path.join, os.path.isdir
+
+    try:
+        # Should be O(1) since it's probably just reading your filesystem journal
+        names = os.listdir(top)
+    except OSError as err:
+        if onerror is not None:
+            onerror(err)
+        return
+
+    dirs, nondirs = [], []
+
+    # O(n) where n = number of files in the directory
+    for name in names:
+        if isdir(join(top, name)):
+            dirs.append(name)
+        else:
+            nondirs.append(name)
+
+    if topdown:
+        yield depth, top, dirs, nondirs
+
+    # Again O(n), where n = number of directories in the directory
+    for name in dirs:
+        new_path = join(top, name)
+        if followlinks or not islink(new_path):
+
+            # Generator so besides the recursive `walk()` call, no additional cost here.
+            for x in walk(new_path, topdown, onerror, followlinks, depth + 1):
+                yield x
+    if not topdown:
+        yield depth, top, dirs, nondirs
diff --git a/gensim/utils.py b/gensim/utils.py
--- a/gensim/utils.py
+++ b/gensim/utils.py
@@ -179,7 +179,8 @@ def copytree_hardlink(source, dest):
         shutil.copy2 = copy2
 
 
-def tokenize(text, lowercase=False, deacc=False, errors="strict", to_lower=False, lower=False):
+def tokenize(text, lowercase=False, deacc=False, encoding='utf8', errors="strict", to_lower=False,
+             lower=False):
     """
     Iteratively yield tokens as unicode strings, removing accent marks
     and optionally lowercasing the unidoce string by assigning True
@@ -195,11 +196,15 @@ def tokenize(text, lowercase=False, deacc=False, errors="strict", to_lower=False
 
     """
     lowercase = lowercase or to_lower or lower
-    text = to_unicode(text, errors=errors)
+    text = to_unicode(text, encoding, errors=errors)
     if lowercase:
         text = text.lower()
     if deacc:
         text = deaccent(text)
+    return simple_tokenize(text)
+
+
+def simple_tokenize(text):
     for match in PAT_ALPHABETIC.finditer(text):
         yield match.group()
 


Test patch:
diff --git a/gensim/test/test_corpora.py b/gensim/test/test_corpora.py
--- a/gensim/test/test_corpora.py
+++ b/gensim/test/test_corpora.py
@@ -8,18 +8,21 @@
 Automated tests for checking corpus I/O formats (the corpora package).
 """
 
+from __future__ import unicode_literals
+
+import codecs
+import itertools
 import logging
 import os.path
-import unittest
 import tempfile
-import itertools
+import unittest
 
 import numpy as np
 
-from gensim.utils import to_unicode
-from gensim.interfaces import TransformedCorpus
 from gensim.corpora import (bleicorpus, mmcorpus, lowcorpus, svmlightcorpus,
                             ucicorpus, malletcorpus, textcorpus, indexedcorpus)
+from gensim.interfaces import TransformedCorpus
+from gensim.utils import to_unicode
 
 # needed because sample data files are located in the same folder
 module_path = os.path.dirname(__file__)
@@ -48,6 +51,10 @@ def __getitem__(self, bow):
 class CorpusTestCase(unittest.TestCase):
     TEST_CORPUS = [[(1, 1.0)], [], [(0, 0.5), (2, 1.0)], []]
 
+    def setUp(self):
+        self.corpus_class = None
+        self.file_extension = None
+
     def run(self, result=None):
         if type(self) is not CorpusTestCase:
             super(CorpusTestCase, self).run(result)
@@ -151,7 +158,7 @@ def test_switch_id2word(self):
             firstdoc = next(iter(corpus))
             testdoc = set((to_unicode(corpus.id2word[x]), y) for x, y in firstdoc)
 
-            self.assertEqual(testdoc, set([('computer', 1), ('human', 1), ('interface', 1)]))
+            self.assertEqual(testdoc, {('computer', 1), ('human', 1), ('interface', 1)})
 
             d = corpus.id2word
             d[0], d[1] = d[1], d[0]
@@ -159,7 +166,7 @@ def test_switch_id2word(self):
 
             firstdoc2 = next(iter(corpus))
             testdoc2 = set((to_unicode(corpus.id2word[x]), y) for x, y in firstdoc2)
-            self.assertEqual(testdoc2, set([('computer', 1), ('human', 1), ('interface', 1)]))
+            self.assertEqual(testdoc2, {('computer', 1), ('human', 1), ('interface', 1)})
 
     def test_indexing(self):
         fname = datapath('testcorpus.' + self.file_extension.lstrip('.'))
@@ -190,7 +197,7 @@ def _get_slice(corpus, slice_):
             return corpus[slice_]
 
         # make sure proper input validation for sliced corpora is done
-        self.assertRaises(ValueError, _get_slice, corpus, set([1]))
+        self.assertRaises(ValueError, _get_slice, corpus, {1})
         self.assertRaises(ValueError, _get_slice, corpus, 1.0)
 
         # check sliced corpora that use fancy indexing
@@ -202,16 +209,16 @@ def _get_slice(corpus, slice_):
 
         # check that TransformedCorpus supports indexing when the underlying
         # corpus does, and throws an error otherwise
+        corpus_ = TransformedCorpus(DummyTransformer(), corpus)
         if hasattr(corpus, 'index') and corpus.index is not None:
-            corpus_ = TransformedCorpus(DummyTransformer(), corpus)
             self.assertEqual(corpus_[0][0][1], docs[0][0][1] + 1)
-            self.assertRaises(ValueError, _get_slice, corpus_, set([1]))
+            self.assertRaises(ValueError, _get_slice, corpus_, {1})
             transformed_docs = [val + 1 for i, d in enumerate(docs) for _, val in d if i in [1, 3, 4]]
             self.assertEquals(transformed_docs, list(v for doc in corpus_[[1, 3, 4]] for _, v in doc))
             self.assertEqual(3, len(corpus_[[1, 3, 4]]))
         else:
             self.assertRaises(RuntimeError, _get_slice, corpus_, [1, 3, 4])
-            self.assertRaises(RuntimeError, _get_slice, corpus_, set([1]))
+            self.assertRaises(RuntimeError, _get_slice, corpus_, {1})
             self.assertRaises(RuntimeError, _get_slice, corpus_, 1.0)
 
 
@@ -323,6 +330,70 @@ def test_load_with_metadata(self):
             doc, metadata = docmeta
             self.assertEqual(metadata[0], i)
 
+    def test_default_preprocessing(self):
+        lines = [
+            "Šéf chomutovských komunistů dostal poštou bílý prášek",
+            "this is a test for stopwords",
+            "zf tooth   spaces   "
+        ]
+        expected = [
+            ['Sef', 'chomutovskych', 'komunistu', 'dostal', 'postou', 'bily', 'prasek'],
+            ['test', 'stopwords'],
+            ['tooth', 'spaces']
+        ]
+
+        corpus = self.corpus_from_lines(lines)
+        texts = list(corpus.get_texts())
+        self.assertEqual(expected, texts)
+
+    def corpus_from_lines(self, lines):
+        fpath = tempfile.mktemp()
+        with codecs.open(fpath, 'w', encoding='utf8') as f:
+            f.write('\n'.join(lines))
+
+        return self.corpus_class(fpath)
+
+    def test_sample_text(self):
+        lines = ["document%d" % i for i in range(10)]
+        corpus = self.corpus_from_lines(lines)
+        corpus.tokenizer = lambda text: text.split()
+        docs = [doc for doc in corpus.get_texts()]
+
+        sample1 = list(corpus.sample_texts(1))
+        self.assertEqual(len(sample1), 1)
+        self.assertIn(sample1[0], docs)
+
+        sample2 = list(corpus.sample_texts(len(lines)))
+        self.assertEqual(len(sample2), len(corpus))
+        for i in range(len(corpus)):
+            self.assertEqual(sample2[i], ["document%s" % i])
+
+        with self.assertRaises(ValueError):
+            list(corpus.sample_texts(len(corpus) + 1))
+
+        with self.assertRaises(ValueError):
+            list(corpus.sample_texts(-1))
+
+    def test_sample_text_length(self):
+        lines = ["document%d" % i for i in range(10)]
+        corpus = self.corpus_from_lines(lines)
+        corpus.tokenizer = lambda text: text.split()
+
+        sample1 = list(corpus.sample_texts(1, length=1))
+        self.assertEqual(sample1[0], ["document0"])
+
+        sample2 = list(corpus.sample_texts(2, length=2))
+        self.assertEqual(sample2[0], ["document0"])
+        self.assertEqual(sample2[1], ["document1"])
+
+    def test_sample_text_seed(self):
+        lines = ["document%d" % i for i in range(10)]
+        corpus = self.corpus_from_lines(lines)
+
+        sample1 = list(corpus.sample_texts(5, seed=42))
+        sample2 = list(corpus.sample_texts(5, seed=42))
+        self.assertEqual(sample1, sample2)
+
     def test_save(self):
         pass
 
@@ -336,6 +407,139 @@ def test_indexing(self):
         pass
 
 
+class TestTextDirectoryCorpus(unittest.TestCase):
+
+    def write_one_level(self, *args):
+        if not args:
+            args = ('doc1', 'doc2')
+        dirpath = tempfile.mkdtemp()
+        self.write_docs_to_directory(dirpath, *args)
+        return dirpath
+
+    def write_docs_to_directory(self, dirpath, *args):
+        for doc_num, name in enumerate(args):
+            with open(os.path.join(dirpath, name), 'w') as f:
+                f.write('document %d content' % doc_num)
+
+    def test_one_level_directory(self):
+        dirpath = self.write_one_level()
+
+        corpus = textcorpus.TextDirectoryCorpus(dirpath)
+        self.assertEqual(len(corpus), 2)
+        docs = list(corpus)
+        self.assertEqual(len(docs), 2)
+
+    def write_two_levels(self):
+        dirpath = self.write_one_level()
+        next_level = os.path.join(dirpath, 'level_two')
+        os.mkdir(next_level)
+        self.write_docs_to_directory(next_level, 'doc1', 'doc2')
+        return dirpath, next_level
+
+    def test_two_level_directory(self):
+        dirpath, next_level = self.write_two_levels()
+
+        corpus = textcorpus.TextDirectoryCorpus(dirpath)
+        self.assertEqual(len(corpus), 4)
+        docs = list(corpus)
+        self.assertEqual(len(docs), 4)
+
+        corpus = textcorpus.TextDirectoryCorpus(dirpath, min_depth=1)
+        self.assertEqual(len(corpus), 2)
+        docs = list(corpus)
+        self.assertEqual(len(docs), 2)
+
+        corpus = textcorpus.TextDirectoryCorpus(dirpath, max_depth=0)
+        self.assertEqual(len(corpus), 2)
+        docs = list(corpus)
+        self.assertEqual(len(docs), 2)
+
+    def test_filename_filtering(self):
+        dirpath = self.write_one_level('test1.log', 'test1.txt', 'test2.log', 'other1.log')
+        corpus = textcorpus.TextDirectoryCorpus(dirpath, pattern="test.*\.log")
+        filenames = list(corpus.iter_filepaths())
+        expected = [os.path.join(dirpath, name) for name in ('test1.log', 'test2.log')]
+        self.assertEqual(expected, filenames)
+
+        corpus.pattern = ".*.txt"
+        filenames = list(corpus.iter_filepaths())
+        expected = [os.path.join(dirpath, 'test1.txt')]
+        self.assertEqual(expected, filenames)
+
+        corpus.pattern = None
+        corpus.exclude_pattern = ".*.log"
+        filenames = list(corpus.iter_filepaths())
+        self.assertEqual(expected, filenames)
+
+    def test_lines_are_documents(self):
+        dirpath = tempfile.mkdtemp()
+        lines = ['doc%d text' % i for i in range(5)]
+        fpath = os.path.join(dirpath, 'test_file.txt')
+        with open(fpath, 'w') as f:
+            f.write('\n'.join(lines))
+
+        corpus = textcorpus.TextDirectoryCorpus(dirpath, lines_are_documents=True)
+        docs = [doc for doc in corpus.getstream()]
+        self.assertEqual(len(lines), corpus.length)  # should have cached
+        self.assertEqual(lines, docs)
+
+        corpus.lines_are_documents = False
+        docs = [doc for doc in corpus.getstream()]
+        self.assertEqual(1, corpus.length)
+        self.assertEqual('\n'.join(lines), docs[0])
+
+    def test_non_trivial_structure(self):
+        """Test with non-trivial directory structure, shown below:
+        .
+        ├── 0.txt
+        ├── a_folder
+        │   └── 1.txt
+        └── b_folder
+            ├── 2.txt
+            ├── 3.txt
+            └── c_folder
+                └── 4.txt
+        """
+        dirpath = tempfile.mkdtemp()
+        self.write_docs_to_directory(dirpath, '0.txt')
+
+        a_folder = os.path.join(dirpath, 'a_folder')
+        os.mkdir(a_folder)
+        self.write_docs_to_directory(a_folder, '1.txt')
+
+        b_folder = os.path.join(dirpath, 'b_folder')
+        os.mkdir(b_folder)
+        self.write_docs_to_directory(b_folder, '2.txt', '3.txt')
+
+        c_folder = os.path.join(b_folder, 'c_folder')
+        os.mkdir(c_folder)
+        self.write_docs_to_directory(c_folder, '4.txt')
+
+        corpus = textcorpus.TextDirectoryCorpus(dirpath)
+        filenames = list(corpus.iter_filepaths())
+        base_names = [name[len(dirpath) + 1:] for name in filenames]
+        expected = [
+            '0.txt',
+            'a_folder/1.txt',
+            'b_folder/2.txt',
+            'b_folder/3.txt',
+            'b_folder/c_folder/4.txt'
+        ]
+        self.assertEqual(expected, base_names)
+
+        corpus.max_depth = 1
+        self.assertEqual(expected[:-1], base_names[:-1])
+
+        corpus.min_depth = 1
+        self.assertEqual(expected[2:-1], base_names[2:-1])
+
+        corpus.max_depth = 0
+        self.assertEqual(expected[2:], base_names[2:])
+
+        corpus.pattern = "4.*"
+        self.assertEqual(expected[-1], base_names[-1])
+
+
 if __name__ == '__main__':
     logging.basicConfig(level=logging.DEBUG)
     unittest.main()
diff --git a/gensim/test/test_miislita.py b/gensim/test/test_miislita.py
--- a/gensim/test/test_miislita.py
+++ b/gensim/test/test_miislita.py
@@ -16,10 +16,9 @@
 from __future__ import with_statement
 
 import logging
+import os
 import tempfile
 import unittest
-import bz2
-import os
 
 from gensim import utils, corpora, models, similarities
 
@@ -45,16 +44,15 @@ def get_texts(self):
         .cor format: one document per line, words separated by whitespace.
 
         """
-        with self.getstream() as stream:
-            for doc in stream:
-                yield [word for word in utils.to_unicode(doc).lower().split()
-                        if word not in CorpusMiislita.stoplist]
+        for doc in self.getstream():
+            yield [word for word in utils.to_unicode(doc).lower().split()
+                    if word not in CorpusMiislita.stoplist]
 
     def __len__(self):
         """Define this so we can use `len(corpus)`"""
         if 'length' not in self.__dict__:
             logger.info("caching corpus size (calculating number of documents)")
-            self.length = sum(1 for doc in self.get_texts())
+            self.length = sum(1 for _ in self.get_texts())
         return self.length
 
 
diff --git a/gensim/test/test_textcorpus.py b/gensim/test/test_textcorpus.py
deleted file mode 100644
--- a/gensim/test/test_textcorpus.py
+++ /dev/null
@@ -1,69 +0,0 @@
-#!/usr/bin/env python
-# -*- coding: utf-8 -*-
-#
-# Copyright (C) 2010 Radim Rehurek <radimrehurek@seznam.cz>
-# Licensed under the GNU LGPL v2.1 - http://www.gnu.org/licenses/lgpl.html
-
-"""
-Automated tests for checking the WikiCorpus
-"""
-
-
-import logging
-import unittest
-
-from gensim.corpora.textcorpus import TextCorpus
-
-
-logger = logging.getLogger(__name__)
-
-
-class TestTextCorpus(unittest.TestCase):
-    # TODO add tests for other methods
-
-    class DummyTextCorpus(TextCorpus):
-        def __init__(self):
-            self.size = 10
-            self.data = [["document%s" % i] for i in range(self.size)]
-
-        def get_texts(self):
-            for document in self.data:
-                yield document
-
-    def test_sample_text(self):
-        corpus = self.DummyTextCorpus()
-
-        sample1 = list(corpus.sample_texts(1))
-        self.assertEqual(len(sample1), 1)
-        self.assertIn(sample1[0], corpus.data)
-
-        sample2 = list(corpus.sample_texts(corpus.size))
-        self.assertEqual(len(sample2), corpus.size)
-        for i in range(corpus.size):
-            self.assertEqual(sample2[i], ["document%s" % i])
-
-        with self.assertRaises(ValueError):
-            list(corpus.sample_texts(corpus.size + 1))
-
-        with self.assertRaises(ValueError):
-            list(corpus.sample_texts(-1))
-
-    def test_sample_text_length(self):
-        corpus = self.DummyTextCorpus()
-        sample1 = list(corpus.sample_texts(1, length=1))
-        self.assertEqual(sample1[0], ["document0"])
-
-        sample2 = list(corpus.sample_texts(2, length=2))
-        self.assertEqual(sample2[0], ["document0"])
-        self.assertEqual(sample2[1], ["document1"])
-
-    def test_sample_text_seed(self):
-        corpus = self.DummyTextCorpus()
-        sample1 = list(corpus.sample_texts(5, seed=42))
-        sample2 = list(corpus.sample_texts(5, seed=42))
-        self.assertEqual(sample1, sample2)
-
-
-if __name__ == '__main__':
-    logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.DEBUG)
-    unittest.main()