alphagov/govspeak — pull request #34 by kalleth (2014-08-12 13:50:03+00:00)
Title: Force character encoding when validating govspeak

When this is released as 2.0.2 it will fix the test failures on master of govuk_content_models.

In prior versions of Kramdown, UTF-8 strings were always returned
regardless of input. However, in the new version, the character encoding
of the input is maintained.

Nokogiri encodes entities differently in order to return a string of the
same encoding as the input data.

For example:

string = "–"

string.force_encoding("UTF-8")
Nokogiri::HTML.parse(string) => "–"

string.force_encoding("US-ASCII")
Nokogiri::HTML.parse(string) => "–"

The Sanitizer, on the other hand, doesn't always return the same
encoding of string as goes in.

This meant that instead of comparing like for like, after the kramdown
upgrade, the outputs from Nokogiri differed. Hence, failing validations.

By forcing the encoding of the input string to `UTF-8`, we avoid issues
with differing character encodings.