ohler55/ox — Issue #6: Encoding issues (Ruby 1.9.x)
Author: dim
Created: 2012-01-17 11:32:48+00:00
State: open

Ox currently doesn't seem to care about (Ruby) encodings. I am not sure where to start this, so here a use case.

Let's create 2 XML documents:

x1 = %(Français).encode("ISO-8859-1")
# => "Fran\xE7ais"
x1.encoding
# => #

x2 = %(Français)
# => "Fran\xC3\xA7ais"
x2.encoding
# => #

With Ox:

class OH < ::Ox::Sax
def start_element(name)
puts "EL: #{name} (#{name.encoding})"
end

def end_element(name)
end

def attr(key, value)
puts "AT: #{key} => #{value} (#{key.encoding} => #{value.encoding})"
end

def text(value)
puts "TX: #{value} (#{value.encoding})"
end
end

::Ox.sax_parse OH.new, StringIO.new(x1)
# => AT: version => 1.0 (US-ASCII => ASCII-8BIT)
# AT: encoding => ISO-8859-1 (US-ASCII => ISO-8859-1)
# EL: tag (US-ASCII)
# AT: key => value (US-ASCII => ISO-8859-1)
# TX: Fran�ais (ISO-8859-1)

::Ox.sax_parse OH.new, StringIO.new(x2)
# => AT: version => 1.0 (US-ASCII => ASCII-8BIT)
# AT: encoding => UTF-8 (US-ASCII => UTF-8)
# EL: tag (US-ASCII)
# AT: key => value (US-ASCII => UTF-8)
# TX: Français (UTF-8)

Now the same with Nokogiri:

class NH
def self.parse(io)
root = Nokogiri::XML(io).root
puts "EL: #{root.name} (#{root.name.encoding})"
root.attributes.each do |key, value|
puts "AT: #{key} => #{value.value} (#{key.encoding} => #{value.value.encoding})"
end
puts "TX: #{root.text} (#{root.text.encoding})"
end
end

NH.parse StringIO.new(x1)
# => EL: tag (UTF-8)
# AT: key => value (UTF-8 => UTF-8)
# TX: Français (UTF-8)

NH.parse StringIO.new(x2)
# => EL: tag (UTF-8)
# AT: key => value (UTF-8 => UTF-8)
# TX: Français (UTF-8)

As you can see, Nokogiri encodes everything correctly to `Encoding.default_external` while Ox's encodings are a little "random".

It gets a lot worse with non-ASCII attributes:
x1 = %(Français).encode("ISO-8859-1")
x2 = %(Français)

NH.parse StringIO.new(x1)
# => EL: tag (UTF-8)
# AT: Português => Español (UTF-8 => UTF-8)
# TX: Français (UTF-8)

NH.parse StringIO.new(x2)
# Same as above

::Ox.sax_parse OH.new, StringIO.new(x1)
# => AT: version => 1.0 (US-ASCII => ASCII-8BIT)
# AT: encoding => ISO-8859-1 (US-ASCII => ISO-8859-1)
# EL: tag (US-ASCII)
# EncodingError: invalid encoding symbol

Any ideas? Cheers, dim