scikit-learn/scikit-learn — pull request #632 by robertlayton (2012-02-15 09:54:02+00:00)
Title: Ignore fix

Fixes issue #629:

When analysing documents, if a bytes array is given to one of the n-gram analyzers, it is decoded using a given charset. However the errors are set to 'ignore', which I feel may hide problems in codes (i.e. using the incorrect encoding may screw with results).

Location: in `feature_extraction.text.py`, lines 156 and 208, for classes WordNGramAnalyzer and CharNGramAnalyzer.

Fix: I'd like to add a keyword to the classes on what to do, default being 'error' rather than 'ignore' (i.e. if people want to ignore errors, they have to explicitly state so).