puppetlabs/puppet — pull request #250 by jeffmccune (2011-12-07 20:38:38+00:00)
Title: (#11246) Fix UTF-8 String#to_yaml bug

commit a01aab2ce758f6b38f3a52c36842834cd08e3644
Author: Jeff McCune
Date: Wed Dec 7 10:45:24 2011 -0800

(#11246) Add UTF-8 String#to_yaml spec tests

This patch adds a specification test to validate that Puppet can
properly serialize a UTF-8 encoded string to YAML. Without this patch
we don't have a test that covers this desired behavior.

This patch provides an automated way to reproduce the bug reported in
ticket #11246.

The specific test for #11246 that will fail with Ruby 1.9 but not with 1.8 are
these two. The tests encode and catch our expectation that UTF-8 encoded
strings properly serialize and de-serialize to and from YAML.

1) UTF-8 String YAML Handling (Bug #11246) UTF-8 String Literal should serialize to YAML
Failure/Error: subject.to_yaml
Encoding::CompatibilityError:
incompatible encoding regexp match (ASCII-8BIT regexp with UTF-8 string)
# ./lib/puppet/util/zaml.rb:241:in `block in to_zaml'
# ./lib/puppet/util/zaml.rb:90:in `first_time_only'
# ./lib/puppet/util/zaml.rb:227:in `to_zaml'
# ./lib/puppet/util/zaml.rb:22:in `dump'
# ./lib/puppet/util/monkey_patches.rb:32:in `to_yaml'
# ./spec/unit/util/zaml_spec.rb:83:in `block (3 levels) in '

2) UTF-8 String YAML Handling (Bug #11246) UTF-8 String Literal should serialize and deserialize to the same thing
Failure/Error: YAML.load(subject.to_yaml).should == subject
Encoding::CompatibilityError:
incompatible encoding regexp match (ASCII-8BIT regexp with UTF-8 string)
# ./lib/puppet/util/zaml.rb:241:in `block in to_zaml'
# ./lib/puppet/util/zaml.rb:90:in `first_time_only'
# ./lib/puppet/util/zaml.rb:227:in `to_zaml'
# ./lib/puppet/util/zaml.rb:22:in `dump'
# ./lib/puppet/util/monkey_patches.rb:32:in `to_yaml'
# ./spec/unit/util/zaml_spec.rb:86:in `block (3 levels) in '

commit b24b89f495071179fff61ffc332c33efb8ea659e
Author: Jeff McCune
Date: Tue Dec 6 22:05:00 2011 -0800

(#11246) Fix UTF-8 String#to_yaml exception

Without this patch, String instances with an encoding of UTF-8 cannot be
serialized to YAML. The method that coverts non-ascii 8bit printable
bytes to escape sequences uses regular expressions to match specific
bytes in the string. This works fine in Ruby 1.8, but Ruby 1.8 adds
encodings to Regexp and String instances. This is a problem because
regular expressions of one encoding may not work with Strings of another
encoding.

This patch fixes the problem by ensuring Regexp's clearly assuming a
byte sequence rather than a character sequence used in the YAML
serialization process are given an ASCII-8BIT encoded string.

A new helper method String#to_ascii8bit is added by this patch to make
Ruby 1.8 support easier. None of this matters with Ruby 1.8 so the
helper method simply returns the string itself. If the String does have
an encoding, but it's already ASCII-8BIT the helper also returns the
string itself.

In the case the String has an encoding and it's not a byte sequence with
ASCII-8BIT encoding, we make a duplicate of the string so we can force
the encoding. The duplicate is necessary so we don't change the
encoding of the original string. For extremely large strings this may
be an issue and other strategies may be necessary.