rubinius/rubinius-compiler — pull request #2 by jemc (2014-04-25 23:07:55+00:00)
Title: Possible fix for Rubinius issue 2997?

https://github.com/rubinius/rubinius/issues/2997

@YorickPeterse, please take a look at this possible fix for 2997. I'm not 100% confident in my knowledge of how encodings work in Ruby to know if I'm fixing it the right way here, but I think I can explain what's going on.

So, compiled_file.rb specifies encoding US-ASCII at the top of the file. This means that everywhere where you call `"".encoding`, it yields `#`.

When marshalling a Tuple, #marshal forces the encoding of each part to `"".encoding`, which is `#`.

When marshalling a String or Symbol, #marshal in its current form will output a marshal string with the same encoding as the source string. (This is the part I changed - now it will force just like Tuple).

When marshalling a CompiledFile, it concatenates all of the marshalled strings of the various components, in order. In the case of the issue 2997, the `marshal(val.name)` (a String/Symbol) was returning a UTF-8 encoded string. When that UTF-8 encoded string was appended to the US-ASCII string `str`, it made `str` become UTF-8. Later, `marshal(val.literals)` (a Tuple) was returning a US-ASCII encoded string that was forced from a string that was sourced from a UTF-8 string. Trying to concatenate this to the UTF-8 string caused the Encoding error.

Again, not sure I fixed this in the right way but I've at least pinpointed what is going on.

Hope to hear some feedback from @YorickPeterse, @brixen, @dbussink.