nltk/nltk — Issue #101: Word segmenter for Chinese text
Author: alexrudnick
Created: 2012-01-17 06:26:06+00:00
State: open

Ordinary Chinese text does not use whitespace to segment characters into words, and therefore the
task of recognizing words is much more challenging than for western languages. There are a variety
of techniques available for carrying out the task, most of which use some machine learning
paradigm. Looking up words in an online Chinese dictionary can be useful, but is never going to be
enough, since many of the words encountered in ordinary text fail to appear in the available
dictionaries. A good overview of the task, with links to training and test data, can be found here:

http://alias-i.com/lingpipe/demos/tutorial/chineseTokens/read-me.html

Migrated from http://code.google.com/p/nltk/issues/detail?id=516