rtomayko/rocco — Issue #73: Unit test for ISO-8859-1 breaks under Ruby 1.9.3
Author: lambda
Created: 2012-01-05 11:01:37+00:00
State: open

When running the unit tests, `test_issue10_utf8_processing` fails when I run it under Ruby 1.9.3.

2) Error:
test_issue10_utf8_processing(RoccoIssueTests):
ArgumentError: invalid byte sequence in UTF-8
/Users/lambda/src/rocco/lib/rocco.rb:235:in `split'
/Users/lambda/src/rocco/lib/rocco.rb:235:in `parse'
/Users/lambda/src/rocco/lib/rocco.rb:134:in `initialize'
/Users/lambda/src/rocco/test/test_reported_issues.rb:31:in `new'
/Users/lambda/src/rocco/test/test_reported_issues.rb:31:in `test_issue10_utf8_processing'

This is because Ruby 1.9 is much more strict about encodings, instead of just reading bytes into the string and hoping you know how to interpret it, it actually keeps track of what encoding it expects the file and the string to be and transcodes between them. This means that we can no longer just read in the ISO-8859-1 as binary, and kind of hope that everything later on ignores it or knows how to cope. Instead, we need to explicitly choose to read the file as ISO-8859-1, and explicitly choose the encoding that we want the string to be in (probably UTF-8 as that's what everyone else will be expecting).