sstephenson/sprockets — pull request #581 by josh (2014-05-31 09:02:19+00:00)
Title: Mime Type Charset detection

Hey @brianmario, this is basically what I was asking for advice on.

The big change is away from a naive `Encoding.default_external` assumption to allowing each content type to define its own charset detection logic.

* `application/javascript` should prefer BOM detect, then fallback to UTF-8
* `text/css` should prefer BOM detect, else read `@charset "";` then fallback to UTF-8
* `text/yaml` should do similar to js
* `text/html` should prefer BOM, else read `` (not implemented here)

Aside from standard unicode BOMs, each content type kinda has its own rules.

In addition to this, Sprockets likes to concatenate things. So in order to work with multiple encoding types, the encoding markers themselves need to be stripped from the content itself and moved to the `.encoding` annotation on the String. This means the decoding function doubles as detection and stripping.

* Does "decoder" make sense as a word for "detect" and "strip"?
* Whats a good name for "text" or "binary"? Right now I'm calling it `:type`, but then I have to say "a mime-type has a type :binary". Don't really like that. Would "mode" work? Maybe thats only for readers.
* Can I implement these byte scanners more efficiently?