scikit-learn/scikit-learn — Issue #656: document classification example is broken
Author: mblondel
Created: 2012-02-28 07:48:27+00:00
State: open

$ python examples/document_classification_20newsgroups.py
[...]

Loading 20 newsgroups dataset for categories:
['alt.atheism', 'talk.religion.misc', 'comp.graphics', 'sci.space']
data loaded
2034 documents (training set)
1353 documents (testing set)
4 categories

Extracting features from the training dataset using a sparse vectorizer
Traceback (most recent call last):
File "examples/document_classification_20newsgroups.py", line 110, in
X_train = vectorizer.fit_transform(data_train.data)
File "/home/mathieu/Desktop/projects/scikit-learn/sklearn/feature_extraction/text.py", line 637, in fit_transform
X = self.tc.fit_transform(raw_documents)
File "/home/mathieu/Desktop/projects/scikit-learn/sklearn/feature_extraction/text.py", line 401, in fit_transform
term_count_current = Counter(self.analyzer.analyze(doc))
File "/home/mathieu/Desktop/projects/scikit-learn/sklearn/feature_extraction/text.py", line 191, in analyze
self.charset_error)
File "/usr/lib/python2.6/encodings/utf_8.py", line 16, in decode
return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xda in position 537: invalid continuation byte