Sage/streamlinejs — pull request #148 by abiyani (2013-06-24 21:42:48+00:00)
Title: _getEncoding() should be less aggressive in returning utf8

This pull request is a proposed fix to _getEncoding() function. Current problems are:
- Behavior is not compliant with recommendations of RFC 2616 (sec-7.2.1).

```
Any HTTP/1.1 message containing an entity-body SHOULD include a
Content-Type header field defining the media type of that body. If
and only if the media type is not given by a Content-Type field, the
recipient MAY attempt to guess the media type via inspection of its
content and/or the name extension(s) of the URI used to identify the
resource. If the media type remains unknown, the recipient SHOULD
treat it as type "application/octet-stream".
```
the function assumes default to be "text/plain", and return "utf8" as the encoding (which is also incorrect for "text/plain" in particular. See below for details).
- Checking if "text"/"json" is a *substring* of content-type to determine utf8 encoding is very fragile and *inaccurate*. For example, a fictitious content-type: "application/I-am-not-text" will be treated as "text". Some real world examples of potentially binary data which will fail on such checks: "application/vnd.oasis.opendocument.text-web", "application/vnd.oasis.opendocument.text", "text/x-fortranapplication/octet-stream" etc. Encoding these streams as "utf8" will lead to irreversible data corruption.
- For a content-type: "blah/foo; charset=XYZ", the function will set encoding as "XYZ". But node's Buffer class can deal with only a very few of the possible encoding (which are actually unlimited, as charset can be any arbitrary string). See below for more details on "charset".

That said, making a general function for determining encoding based on content-type (unless a "charset" is explicitly given) is an incredibly hard problem. I will outline some of the reasons:
- Different "Content-Type" can have arbitrary rules to determine default encoding, i.e., when charset is missing. Examples:
- ```text/plain``` (in context of MIME): http://tools.ietf.org/html/rfc2046#section-4.1.2, says default charset must be assumed "US-ASCII".
- ```application/json```: The correct way is to check bit pattern of first 2 bytes of the content body and determine if it's UTF-32BE, UTF-16BE, etc. (RFC-4627-section-3)
- "Content-encoding" header can indicate additional content encoding applied to data (which is usually the case when data is compressed). Ref: RFC-2616(sec 7.2.1)

Even when a "charset" is present, it's allowed to be an arbitrary string (but recommended to be one of the values registered with IANA: http://www.iana.org/assignments/character-sets/character-sets.xml). However Node's Buffer class can only deal with a handful of them (http://nodejs.org/api/buffer.html#buffer_buffer).

I have taken a shot at making a minimum viable _getEncoding() function. The main idea is that we return a particular encoding only if we are absolutely sure that it is the desired encoding, and do not try to guess anything ourselves. I think it's much better to leave encoding to downstream code, then set incorrect encoding ourselves (which will lead to irreversible data corruption).

**Feature Request:**
Ability to pass an extra option to ```HttpServerRequest()/HttpClientResponse()``` to disable the auto-setting of encoding (irrespective of the Content-Type header). IMHO, it will be better to follow node's behavior, and set default to no encoding always (but that will break existing code, who rely on _getEncoding() to do this task for them). This is useful in variety of cases, specifically when you just want to pump raw bytes in/out without encoding.