use nltk corpora

Squashed commit of the following:

commit f3307f8208d0f35fc5522eb1f3599d8523632a78
Author: Conor Schaefer <conor.schaefer@gmail.com>
Date:   Thu Jun 19 22:15:13 2014 -0400

    updates README

    Squashed commit of the following:

    commit 501955ed41103dc6ea7e2344603fff251c754c9c
    Author: Conor Schaefer <conor.schaefer@gmail.com>
    Date:   Thu Jun 19 22:10:43 2014 -0400

        removes old info

    commit 76104c7c18a83a8bd14d38b91ed74d11e16c4f45
    Author: Conor Schaefer <conor.schaefer@gmail.com>
    Date:   Thu Jun 19 22:09:50 2014 -0400

        updates bootstrap cmd output

    commit 49e61a4ce8f8abc7ddc4cc3dc26cdc9de4b9bc8c
    Author: Conor Schaefer <conor.schaefer@gmail.com>
    Date:   Thu Jun 19 22:00:47 2014 -0400

        reduces steps

commit 6086c95a49e2b5e2efd6fc26f5ed9390417dc660
Author: Conor Schaefer <conor.schaefer@gmail.com>
Date:   Thu Jun 19 21:59:17 2014 -0400

    preloads multiple nltk corpora to build combined lm

    Squashed commit of the following:

    commit 7504c7c853a26025dbb81deef7e30f68a1fbc22e
    Author: Conor Schaefer <conor.schaefer@gmail.com>
    Date:   Thu Jun 19 21:58:50 2014 -0400

        uses whitelist to select corpora

        Squashed commit of the following:

        commit 9b07b465bcf17873e9cc5a44cb5f53a93e3a87e4
        Author: Conor Schaefer <conor.schaefer@gmail.com>
        Date:   Thu Jun 19 21:57:58 2014 -0400

            even less verbosity

        commit 3f9d90e38296e712d9791c93a20af2006f181730
        Author: Conor Schaefer <conor.schaefer@gmail.com>
        Date:   Thu Jun 19 21:45:37 2014 -0400

            cleaner one-line printing

        commit ea74044b7e12a1aa25e10c274569e4c33d25875a
        Author: Conor Schaefer <conor.schaefer@gmail.com>
        Date:   Thu Jun 19 21:39:01 2014 -0400

            clean one-line output

        commit 61af127dffdf136ee956cac7ab7a9912cb10ec30
        Author: Conor Schaefer <conor.schaefer@gmail.com>
        Date:   Thu Jun 19 21:29:07 2014 -0400

            saner verbosity

        commit b7638231f1d29d64e5e0fe888eb6c42e45299ce8
        Author: Conor Schaefer <conor.schaefer@gmail.com>
        Date:   Thu Jun 19 21:22:34 2014 -0400

            adjusts whitelist

        commit 560b7912686e856e625424f8c9d58ddaf12aa43e
        Author: Conor Schaefer <conor.schaefer@gmail.com>
        Date:   Thu Jun 19 21:18:37 2014 -0400

            removes unused def

        commit 7b4c665945a4b78d132ff17d8cae55d5cf1d3c7e
        Author: Conor Schaefer <conor.schaefer@gmail.com>
        Date:   Thu Jun 19 21:16:49 2014 -0400

            removes old logic

        commit bdfd5cbee87eb9cda356da1e6a12ff1b6e0cd474
        Author: Conor Schaefer <conor.schaefer@gmail.com>
        Date:   Thu Jun 19 21:12:44 2014 -0400

            first draft of whitelist

    commit d796a76992bac1d7cd7519106a6d8b285c5a28fb
    Author: Conor Schaefer <conor.schaefer@gmail.com>
    Date:   Thu Jun 19 20:35:44 2014 -0400

        catch unicode exceptions in file writing

    commit 98273fad54feced3a1ca674c1f634830684df61e
    Author: Conor Schaefer <conor.schaefer@gmail.com>
    Date:   Thu Jun 19 20:26:43 2014 -0400

        tries to filter out ASCII files (for now)

    commit bb5c65c2128277f341347ea45a32e9a893ae0f8e
    Author: Conor Schaefer <conor.schaefer@gmail.com>
    Date:   Thu Jun 19 20:13:13 2014 -0400

        simpler corpus namefinding

    commit 0fc4be3b946a2c60cab7bab1ade6256a265c6876
    Author: Conor Schaefer <conor.schaefer@gmail.com>
    Date:   Thu Jun 19 20:04:47 2014 -0400

        checks for .sents() method

    commit f33efd6cbe7f5888583ac452634bae0236fffd26
    Author: Conor Schaefer <conor.schaefer@gmail.com>
    Date:   Thu Jun 19 19:31:38 2014 -0400

        fixes file encoding

        Squashed commit of the following:

        commit ddea7d4f350025606af70339eccfe1256c012537
        Author: Conor Schaefer <conor.schaefer@gmail.com>
        Date:   Thu Jun 19 19:31:20 2014 -0400

            fixes repr

        commit 1fe49b9aea2da35747ebfdb968d0990a27e0b02c
        Author: Conor Schaefer <conor.schaefer@gmail.com>
        Date:   Thu Jun 19 19:24:14 2014 -0400

            adds comments

        commit 80d2d2436e8517b1a18dae887defe1f54e5ab217
        Author: Conor Schaefer <conor.schaefer@gmail.com>
        Date:   Thu Jun 19 19:20:11 2014 -0400

            fixes encoding on lm file

        commit 7b0b956d742e6a33c23d0854dcd77a9e53f3df1e
        Author: Conor Schaefer <conor.schaefer@gmail.com>
        Date:   Thu Jun 19 19:13:37 2014 -0400

            first attempt, failing

    commit 73a6ed2ec800a344c3bf08e30a5a4c4ea9f10963
    Author: Conor Schaefer <conor.schaefer@gmail.com>
    Date:   Mon Jun 16 01:37:08 2014 -0400

        fixes filepaths for single LM

        still getting Unicode errors, this time in importNgrams...

    commit b68cd495a902536154b5fbaae42d547bb39ede8c
    Author: Conor Schaefer <conor.schaefer@gmail.com>
    Date:   Mon Jun 16 01:30:25 2014 -0400

        generates combined LM

    commit aa762f974fb8e8a6b16e12cbb00454eda817bf96
    Author: Conor Schaefer <conor.schaefer@gmail.com>
    Date:   Mon Jun 16 01:17:55 2014 -0400

        fixes overly aggressive corpora filtering

    commit 8a346607fdaf82a0ac5a1cd35f0d7404de662b0e
    Author: Conor Schaefer <conor.schaefer@gmail.com>
    Date:   Mon Jun 16 01:11:15 2014 -0400

        returns corpora as list, only ones with .sents()

    commit 6f5b8619e1dd2209dc9aeff75d7e1be67cd87917
    Author: Conor Schaefer <conor.schaefer@gmail.com>
    Date:   Mon Jun 16 00:55:13 2014 -0400

        adds manual blacklist for incompatible corpora

    commit a1e3ebc6790c781baaadc2a37d57c1ab849d3422
    Author: Conor Schaefer <conor.schaefer@gmail.com>
    Date:   Mon Jun 16 00:19:53 2014 -0400

        builds multiple LMs

commit b5cc0471cad3ade75b9a987715c43796c2fc589a
Author: Conor Schaefer <conor.schaefer@gmail.com>
Date:   Sun Jun 15 23:57:25 2014 -0400

    separates "counts" and "sentences" files for corpora

    Squashed commit of the following:

    commit 4618816e1274add950a761d713532c6d0cb44871
    Author: Conor Schaefer <conor.schaefer@gmail.com>
    Date:   Sun Jun 15 23:55:47 2014 -0400

        makes importNgrams more modular

        Squashed commit of the following:

        commit ac8c2a67168c6f4e545339974869c961f1236132
        Author: Conor Schaefer <conor.schaefer@gmail.com>
        Date:   Sun Jun 15 23:54:06 2014 -0400

            separate counts and sentences

        commit 57463af5ff7eea6ed5627ad584d292417caba52b
        Author: Conor Schaefer <conor.schaefer@gmail.com>
        Date:   Sun Jun 15 23:45:01 2014 -0400

            fixes lm generation by making calls blocking

        commit b7fe69c4aab16afd8b1fe5370df0d82c5fff5d98
        Author: Conor Schaefer <conor.schaefer@gmail.com>
        Date:   Sun Jun 15 20:59:46 2014 -0400

            fixes loading count display

        commit e6ff6365fefa5b83817020464d6d8191959174a4
        Author: Conor Schaefer <conor.schaefer@gmail.com>
        Date:   Sun Jun 15 20:45:20 2014 -0400

            adds comments

        commit 3b54b47fec4ac0b5ee7e369de5e12458dcd39a36
        Author: Conor Schaefer <conor.schaefer@gmail.com>
        Date:   Sun Jun 15 20:35:03 2014 -0400

            breaks preparation logic into defs

    commit c25fd381be1574f289d6f534473f49af04e5cb47
    Author: Conor Schaefer <conor.schaefer@gmail.com>
    Date:   Sun Jun 15 19:14:25 2014 -0400

        move tmp files aside

    commit 4a72371448306c6e79ac2e05b71df539ec67ed00
    Author: Conor Schaefer <conor.schaefer@gmail.com>
    Date:   Sun Jun 15 19:09:23 2014 -0400

        removes old comment

    commit aed1b5899c71a139d750669188452cf37d0577f3
    Author: Conor Schaefer <conor.schaefer@gmail.com>
    Date:   Sun Jun 15 18:36:42 2014 -0400

        adds corpus and fixtures dirs

commit 9d49d61efb4eb45bc6e218e377c49c4e532ba6a0
Author: Conor Schaefer <conor.schaefer@gmail.com>
Date:   Sun Jun 15 18:34:34 2014 -0400

    creates utils dir

commit a382430d793c77c60bfad11e8b650f04f3e4d665
Author: Conor Schaefer <conor.schaefer@gmail.com>
Date:   Sun Jun 15 16:53:11 2014 -0400

    converts print statements to stdout flushes

commit ab948f14e062a7f1f55ab1f3d12895df1ae14d8e
Author: Conor Schaefer <conor.schaefer@gmail.com>
Date:   Sun Jun 15 15:27:55 2014 -0400

    imports proper PATH for SRILM binaries

commit a17467bc73247096a64356378e115c56dfeeb2ad
Author: Conor Schaefer <conor.schaefer@gmail.com>
Date:   Sat Jun 14 22:52:14 2014 -0400

    adds nltk to requirements

commit 78f4c9e49cbab781157036b0e46d14e440d20ef4
Author: Conor Schaefer <conor.schaefer@gmail.com>
Date:   Sat Jun 14 22:41:03 2014 -0400

    cleans up bootstrap script

commit fa8edd09e18beb953426efe221b9049ad247b185
Author: Conor Schaefer <conor.schaefer@gmail.com>
Date:   Sat Jun 14 22:38:53 2014 -0400

    adds NGRAM_ORDER var to settings

commit 21ecad9f52cf28f51505662495c11de54ca8c0e0
Author: Conor Schaefer <conor.schaefer@gmail.com>
Date:   Sat Jun 14 22:34:42 2014 -0400

    fixes relative paths

commit 1afc15d0ac4037c0d3266959b8921f765f658fd9
Author: Conor Schaefer <conor.schaefer@gmail.com>
Date:   Sat Jun 14 22:34:29 2014 -0400

    adds srilm logic for preparing corpora

commit f81c493d32987add91075cdf55e173c1a63d0968
Author: Conor Schaefer <conor.schaefer@gmail.com>
Date:   Sat Jun 14 22:30:14 2014 -0400

    starts adding srilm logic to corpus preparation

commit b7c08067fa439aeeb1c4d699a37c69563bef8a23
Author: Conor Schaefer <conor.schaefer@gmail.com>
Date:   Sat Jun 14 22:26:45 2014 -0400

    adds rough script for preparing corpora

commit d442ebfecd7757740968f09aed6199bcb7504ef5
Author: Conor Schaefer <conor.schaefer@gmail.com>
Date:   Sat Jun 14 22:18:54 2014 -0400

    prettier corpora downloading

commit 6af61a49140b88a3a68652c7c4c0f601b9bd73c3
Author: Conor Schaefer <conor.schaefer@gmail.com>
Date:   Sat Jun 14 19:32:43 2014 -0400

    fixes relative paths

commit 44a3cff0779c8707c7b666d35d9933270d73f4e5
Author: Conor Schaefer <conor.schaefer@gmail.com>
Date:   Wed Jun 4 17:40:10 2014 -0400

    adds simple downloading command to manage.py