Empact/roo — pull request #8 by aew (2012-12-31 02:36:00+00:00)
Title: Common CSV encoding errors

Hi -

Pull request not intended to merge - rather to discuss encoding errors.

I'm not sure what your experience has been, but I need to take files generated by non-technical users on both mac and pc, and I constantly have "invalid UTF-8 byte sequence" errors, and slightly less frequent "Malformed CSV" due to an unclosed quote or something.

In general, I'd rather brute force things and get on with it, as it's impractical for me to ask users to fix.

This isn't appropriate for every use case (it is UTF-8, strips quotes, and may have performance implications for a start), but I've read about enough people struggling with this that perhaps a "brute_force_csv" flag could be incorporated?

Also, despite the deprecation warnings from iconv, it's the only method I've found that reliably forces encoding - somehow string#encode / force_encoding still throws the invalid UTF byte sequence error.

This pull request worked for me to get roo to parse a file properly that is was previously choking on.

Cheers!