Perl-Toolchain-Gang/YAML-Tiny — pull request #2 by dagolden (2013-09-29 03:00:57+00:00)
Title: Fix broken UTF-8 handling

This PR adds a test that exposes suboptimal UTF-8 handling and fixes the following:

* `read_string` was apparently designed to take anything and cope with it, but it's heuristics were insufficient for strings already decoded to UTF-8, particularly invalid ones from readling latin1 on a :utf8 handle. The PR refactors and improves the heuristics.

* `write` didn't write UTF-8 encoded files. The output was whatever perl did, either downgraded (latin1) if there were no wide characters or UTF-8 if there were wide characters. The PR adds a `write_utf8_string` method and has the `write` method use that.

* The arguably poorly-defined `write_string` method is kept as is, but documented it as returning characters.

* Minimal Perl required for Unicode support is bumped to v5.8.1 for utf8::is_utf8

I believe these changes should address most outstanding issues. Code using `write` automatically gets consistent, correct output. Code using `write_string` and doing its own UTF-8 encoding will continue to work. Code using `write_string` and writing without UTF-8 encoding can switch to `write_utf8_string` (and bump the dependency) to get correct behavior.

This PR does not address the fact that Dump follows the YAML.pm convention of returning characters instead of the YAML::XS convention of returning UTF-8 encoded octets. I have emailed various YAML-involved parties for their thoughts on how to address this API divergence and can amend this PR if a consensus emerges.

Note: commit c2ea863376fd6aadc515c06d1713f2ddfe798712 breaks tests until commit cac5c092312150df2612b4f7b5599b157a06d20f. After code review, I recommend squashing into a single commit, but am curious what others think about that.