rtomayko/tilt — pull request #175 by judofyr (2013-03-02 22:17:48+00:00)
Title: Implement minimal encoding support

Oh boy, it's this time again. I want to fix Tilt's encoding support so that it's at least *usable*. This PR tries to handle encodings correctly within Tilt. I've excluded transcoding because it's fairy simple to implement this *outside* of Tilt.

## The encoding of `data`

The encoding of the data is done in two "steps". First we try to *guess* the encoding, then we check if the Template-implementation wants to override it.

Step 1: If you're using a custom reader (`Template.new { foobar }`) we already know the encoding. If you're reading a file, then we read it as binary and force_encoding it to `default_external` (to avoid raising exceptions).

Step 2: There's a method called `#default_encoding`. If this returns a truthy value, we force_encode the data to this encoding. By default it simply looks up the `:default_encoding`-option, but template engines can override this with custom behavior (e.g. always return UTF-8 for CoffeeScript)

This means that the encoding of the `data` has/is:

* Sensible defaults (Encoding.default_external)
* Overridable by the user (through the `:default_encoding`-option)
* Overridable by the template engine (through `#default_encoding)

## The encoding of Tilt's generated source code

Tilt combines the output from the template engine with its own "add local variables and define it as an unbound method for performance". It's the encoding of this source code that determines the encoding that the code runs under (e.g. the value of `"".encoding`).

In this PR we use the same encoding for Tilt's source code as the template engines source code (result from `precompiled_template`). For legacy reasons I've added support for extracting magic comments, so Tilt behaves the same wether a template engine returns `"foo".force_encoding('…')` or `"# coding: …\nfoo` (although the former is preferred)

The encoding of the final string will (most likely) be the same as the source code encoding (although technically the source code can return whatever string with whatever encoding it likes).

## Thoughts?

Is this good enough for now? Will this solve your problems?

/cc @josh, @rtomayko, @mislav, @djanowski, @rue, @nesquena, @DAddYE, @apotonick, @brianmario, @rkh, @apohllo, @argent-smith, @fibric