nltk/nltk — Issue #129: BrillTagger fails on non-ascii text
Author: alexrudnick
Created: 2012-01-17 06:26:44+00:00
State: open

What version of NLTK are you using? (See nltk.__version__). Please only
submit bug reports for the current version.
2.0b9

What steps will reproduce the problem? (e.g. include Python source code)
Get https://github.com/japerk/nltk-trainer and run "python train_tagger.py --brill mac_morpho"

What is the expected output? What do you see instead?
A successfully trained tagger, with no errors. Instead, get the following output with an exception at the end:
loading nltk.corpus.mac_morpho
51397 tagged sents, training on 51397
training AffixTagger with affix -3 and backoff <DefaultTagger: tag=-None->
training <class 'nltk.tag.sequential.UnigramTagger'> tagger with backoff <AffixTagger: size=7476>
training <class 'nltk.tag.sequential.BigramTagger'> tagger with backoff <UnigramTagger: size=26085>
training <class 'nltk.tag.sequential.TrigramTagger'> tagger with backoff <BigramTagger: size=15708>
Training Brill tagger on 51397 sentences...
Finding initial useful rules...
Found 64434 useful rules.
Selecting rules...
Traceback (most recent call last):
File "train_tagger.py", line 227, in <module>

File "/home/jacob/nltk-trainer/nltk_trainer/tagging/training.py", line 13, in train_brill_tagger
return trainer.train(train_sents, **kwargs)
File "/usr/local/lib/python2.6/dist-packages/nltk/tag/brill.py", line 871, in train
rule = self._best_rule(train_sents, test_sents, min_score)
File "/usr/local/lib/python2.6/dist-packages/nltk/tag/brill.py", line 1017, in _best_rule
best_rules.sort(key=repr)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe3' in position 33: ordinal not in range(128)

It appears that the brill tagger sorts rules based on the repr() of the rule, which apparently is assumed to be ascii. I hope there's an equivalent method of sorting that does not require an ascii repr.

This may also be a corpus issue, as I've had no problems training on many of the other corpora, including sinica_treebank and floresta.

Migrated from http://code.google.com/p/nltk/issues/detail?id=610