nltk/nltk — Issue #107: Language ID module
Author: alexrudnick
Created: 2012-01-17 06:26:14+00:00
State: open

It would be nice to have a language ID module in NLTK. There are lots of
free implementations out there, most recycling the same couple of ideas:
http://www.let.rug.nl/~vannoord/TextCat/competitors.html

(including some in Python).

I am willing to donate character three-gram data for about 450 languages
assembled by the web crawler I've been running, here:

http://borel.slu.edu/crubadan/index.html

Table of cosine distances:
http://borel.slu.edu/crubadan/table.html

Migrated from http://code.google.com/p/nltk/issues/detail?id=532

earlier comments

StevenBird1 said, at 2010-04-16T21:09:55.000Z:

The language ID data has been added to NLTK's corpus collection
(nltk_data/corpora/langid), currently item #25 in NLTK's corpus list
(http://nltk.googlecode.com/svn/trunk/nltk_data/index.xml). We need a corpus reader
that loads the data for an individual language/writing-system into a FreqDist. We
might also want programmatic access to the code table (table.txt), which contains the
mapping between the internal identifiers and ISO 639-3 codes.