jekyll/jekyll — pull request #2420 by albertogg (2014-05-18 00:27:36+00:00)
Title: Encode URLs in utf-8 when escaping and unescaping

I've been doing some tests to try to fix issue #2379 after reading @fabianrbz [comment](https://github.com/jekyll/jekyll/issues/2379#issuecomment-43285780). In particular I analyzed what *was* the encoding behavior when using `CGI.escape(path)`, what *is* the behavior with `URI.escape(path, /[^a-zA-Z\d\-._~!$&\'()*+,;=:@\/]/)`. I also tested how the **Addressable** gem worked using `Addressable::URI.escape(path)`, none the less I read a bit about [RFC 3986](http://tools.ietf.org/html/rfc3986) and all of this led me to this pull request.

Here are the examples:

**What Jekyll is doing right now:** It gets a *UTF-8* string, when it gets escaped, the string is converted to *ASCII*, but when it is unescaped it remains *ASCII* and this is when the problem occurs. We can easily fixit by encoding the string to *UTF-8* prior unescaping it.
```ruby
# require 'uri'

pry(main)> uri_uri = 'http://localhost:4000/rails%E7%AC%94%E8%AE%B0/2014/04/20/go-to-top-button-in-rails/'
#=> "http://localhost:4000/rails%E7%AC%94%E8%AE%B0/2014/04/20/go-to-top-button-in-rails/"
uri_uri.encoding
#=> #
pry(main)> uri_escape = URI.escape(uri_uri, /[^a-zA-Z\d\-._~!$&\'()*+,;=:@\/]/)
#=> "http://localhost:4000/rails%25E7%25AC%2594%25E8%25AE%25B0/2014/04/20/go-to-top-button-in-rails/"
pry(main)> uri_escape.encoding
#=> #
pry(main)> uri_unescape = URI.unescape(uri_escape)
#=> "http://localhost:4000/rails%E7%AC%94%E8%AE%B0/2014/04/20/go-to-top-button-in-rails/"
pry(main)> uri_unescape.encoding
#=> #
pry(main)> uri_unescape = URI.unescape(uri_escape.encode('utf-8'))
#=> "http://localhost:4000/rails%E7%AC%94%E8%AE%B0/2014/04/20/go-to-top-button-in-rails/"
pry(main)> uri_unescape.encoding
#=> #
pry(main)> uri_escape = URI.escape(uri_uri, /[^a-zA-Z\d\-._~!$&\'()*+,;=:@\/]/).encode('utf-8')
#=> "http://localhost:4000/rails%25E7%25AC%2594%25E8%25AE%25B0/2014/04/20/go-to-top-button-in-rails/"
pry(main)> uri_escape.encoding
#=> #
pry(main)> uri_unescape = URI.unescape(uri_escape)
#=> "http://localhost:4000/rails%E7%AC%94%E8%AE%B0/2014/04/20/go-to-top-button-in-rails/"
pry(main)> uri_escape.encoding
#=> #
```

**This is what Jekyll *was* doing with CGI:** when it gets a *UTF-8* string and escapes it or unescapes it, the string remained *UTF-8*. If it gets an *ASCII* string it will escape it as an *ASCII* string, but when unescapes it, it will change it to *UTF-8*. So that why the problem never existed in Jekyll < 2.0.0.
```ruby
# require 'cgi'

pry(main)> cgi_uri = 'http://localhost:4000/rails%E7%AC%94%E8%AE%B0/2014/04/20/go-to-top-button-in-rails/'.encode(Encoding::ASCII)
#=> "http://localhost:4000/rails%E7%AC%94%E8%AE%B0/2014/04/20/go-to-top-button-in-rails/"
pry(main)> cgi_uri.encoding
#=> #
pry(main)> cgi_escape = CGI.escape(cgi_uri)
#=> "http%3A%2F%2Flocalhost%3A4000%2Frails%25E7%25AC%2594%25E8%25AE%25B0%2F2014%2F04%2F20%2Fgo-to-top-button-in-rails%2F"
pry(main)> cgi_escape.encoding
#=> #
pry(main)> cgi_unescape = CGI.unescape(cgi_escape)
#=> "http://localhost:4000/rails%E7%AC%94%E8%AE%B0/2014/04/20/go-to-top-button-in-rails/"
pry(main)> cgi_unescape.encoding
#=> #

pry(main)> cgi_uri = 'http://localhost:4000/rails%E7%AC%94%E8%AE%B0/2014/04/20/go-to-top-button-in-rails/'
#=> "http://localhost:4000/rails%E7%AC%94%E8%AE%B0/2014/04/20/go-to-top-button-in-rails/"
pry(main)> cgi_uri.encoding
#=> #
pry(main)> cgi_escape = CGI.escape(cgi_uri)
#=> "http%3A%2F%2Flocalhost%3A4000%2Frails%25E7%25AC%2594%25E8%25AE%25B0%2F2014%2F04%2F20%2Fgo-to-top-button-in-rails%2F"
pry(main)> cgi_escape.encoding
#=> #
pry(main)> cgi_unescape = CGI.unescape(cgi_escape)
#=> "http://localhost:4000/rails%E7%AC%94%E8%AE%B0/2014/04/20/go-to-top-button-in-rails/"
pry(main)> cgi_unescape.encoding
#=> #
```

**Addressable gem** Always converts to *UTF-8* when escaping or unescaping, so the problem never existed.
```ruby
# require 'addressable'

pry(main)> ad_uri = 'http://localhost:4000/rails%E7%AC%94%E8%AE%B0/2014/04/20/go-to-top-button-in-rails/'.encode(Encoding::ASCII)
#=> "http://localhost:4000/rails%E7%AC%94%E8%AE%B0/2014/04/20/go-to-top-button-in-rails/"
pry(main)> ad_uri.encoding
#=> #
pry(main)> ad_escape = Addressable::URI.escape(ad_uri)
#=> "http://localhost:4000/rails%25E7%25AC%2594%25E8%25AE%25B0/2014/04/20/go-to-top-button-in-rails/"
pry(main)> ad_escape.encoding
#=> #

pry(main)> ad_escape_ascii = ad_escape.force_encoding(Encoding::ASCII)
=> "http://localhost:4000/rails%25E7%25AC%2594%25E8%25AE%25B0/2014/04/20/go-to-top-button-in-rails/"
pry(main)> ad_escape_ascii.encoding
#=> #
pry(main)> ad_unescape = Addressable::URI.unescape(ad_escape_ascii)
#=> "http://localhost:4000/rails%E7%AC%94%E8%AE%B0/2014/04/20/go-to-top-button-in-rails/"
pry(main)> ad_unescape.encoding
#=> #
```

Wow, this was really hard for me to write. I hope everyone understand what I was trying to express.