AcademicWorks/eightball — pull request #1 by ouengineer (2013-12-31 22:58:26+00:00)
Title: This may be all wrong...

Sorry if this ends up being a waste of time for you guys to review. But, I've noticed a few encoding related issues pop up and decided to take a fresh look at Eightball (read: naive).

The problems I am trying to address with this change is 1.) ensure that the BOM removal code is executed and 2.) fix an issue whereby only non-UTF-8 encoded strings were getting cleaned up.

This may be an overly simplistic approach. The idea behind the changes is to first check to see if the string is validly encoded and UTF-8. If not, force it to UTF-8 via #force_encode. Then, actually encode the string to UTF-16. This helps to help massage invalid encoded strings to be coerced into a proper encoding. Then encode the string again to UTF-8. Finally, remove the BOM, if it exist. I'm also replacing invalid and undefined characters with '?' as opposed to the default.

Thanks for the input!