ddollar/valkyrie — pull request #5 by whatcould (2012-12-29 15:35:28+00:00)
Title: Add option to force encoding

Hi,

TLDR: I added a third option to the CLI, which forces encoding of the data from the first database using String#encode(encoding, {:invalid => :replace, :undef => :replace}). Second commit just adds information on this change to the README.

==

I had all sorts of character encoding issues with an old MySQL database. All tables were marked as UTF-8, but for some reason the strings coming across were not always UTF-8, and I would get this error:

PG::Error: ERROR: invalid byte sequence for encoding "UTF8": 0x87 (Sequel::DatabaseError)

In my research I discovered:

1. the mysql gem, for some reason, was passing all strings as ASCII-8BIT. ( as reported by the Ruby 1.9 String#encoding method: column_data.encoding)
2. with the mysql2 gem, **almost** all strings were UTF8, but some strings were inexplicably still ASCII.

(btw, valkyrie appears to be Ruby 1.9 only; might be worth noting that in the Readme, since people upgrading to Postgres might also be on the road to upgrade old apps from Ruby 1.8.)