rgrove/sanitize — Issue #49: Bad encoding on non-ASCII replacements in transformer
Author: Zert
Created: 2011-12-02 04:13:01+00:00
State: open

Hello

I have strange behaviour of transformer. When I trying to replace node content with non-ascii strings, sanitizer returns double-encoded to utf-8 string:

This is my transformer:

```ruby
lambda do |env|
node = env[:node]
node_name = env[:node_name]
return if node_name == 'pre'
return if node_name == 'code'
if !node.text?
#text = node.text.to_s
text = "тест"
#text.gsub!(/\n[\n]+/u, "

")
#text.gsub!(/\n/u, "
")
node.replace(text)
end
{:node_whitelist => [node]}
end
```
IO Code:

```ruby
intext = STDIN.read()
outtext = Sanitize.clean(intext, HTMLSanitize::Config::Custom)
outlen = outtext.size()
STDOUT.write(outtext)
```

Output of sanitizer:

$ ./priv/html_sanitizer.rb

тест
тест

ÑÐµÑÑ

But I expect `тест` output
How I can solve this problem with encodings?

Thank you