scikit-learn/scikit-learn — Issue #629: 'ignore' for Unicode conversion in {Word,Char}NGramAnalyzer
Author: robertlayton
Created: 2012-02-15 03:55:17+00:00
State: open

When analysing documents, if a bytes array is given to one of the n-gram analyzers, it is decoded using a given charset. However the errors are set to 'ignore', which I feel may hide problems in codes (i.e. using the incorrect encoding may screw with results).

Location: in `feature_extraction.text.py`, lines 156 and 208, for classes WordNGramAnalyzer and CharNGramAnalyzer.

Fix: I'd like to add a keyword to the classes on what to do, default being 'error' rather than 'ignore' (i.e. if people want to ignore errors, they have to explicitly state so).