nltk/nltk — Issue #122: nltk.corpus.stopwords should return unicode strings
Author: alexrudnick
Created: 2012-01-17 06:26:35+00:00
State: open

The stopword lists are UTF-8 encoded strings:

>>> nltk.corpus.stopwords.words("swedish") [14:18]
['f\xc3\xb6r', 's\xc3\xa5', 'till', '\xc3\xa4r']

>>> nltk.corpus.stopwords.words("russian") [3:7]
['\xd0\xbd\xd0\xb5', '\xd1\x87\xd1\x82\xd0\xbe', '\xd0\xbe\xd0\xbd', '\xd0\xbd\xd0\xb0']

Much better would be to return them in unicode:

[u'f\xf6r', ...]

[u'\u043d\u0435', ...]

Migrated from http://code.google.com/p/nltk/issues/detail?id=583