brianmario/charlock_holmes — Issue #6: How to transcode and write file as utf-8? (Ruby 1.8.7)
Author: nathany
Created: 2012-01-08 05:43:01+00:00
State: open

Hey,

I'm trying to transcode some CSV files but it ends up being gobbly gook. Charlock is detecting the file as ISO-8859-1, whereas BBEdit says the same file is MacRoman (Western), as exported from the Mac version of Excel.

```ruby
def transcode(filename)
require 'charlock_holmes'

content = File.read(filename)
detection = ::CharlockHolmes::EncodingDetector.detect(content)
puts "Unknown encoding for #{filename}" and return if detection.nil?

encoding = detection[:encoding]
return if encoding == 'UTF-8' # already what we want

puts "Transcoding from #{encoding} to UTF-8 for #{filename}"
utf8_encoded_content = CharlockHolmes::Converter.convert(content, encoding, 'UTF-8')
File.open(filename, 'w:utf-8') {|f| f.write(utf8_encoded_content) }
end
```

Is there a way to tell if this is an issue with ICU's detection vs. something I'm doing wrong?

Thanks, Nathan.