dispatch/reboot — pull request #72 by eed3si9n (2013-12-16 21:12:28+00:00)
Title: POSTing a String via <<(String) defaults to UTF-8 encoded "text/plain"

## the `<<` encoding issue

The document says:

> You can also POST an arbitrary string:
>
> ```scala
> def myPostWithBody = myRequest << """{"key": "value"}"""
> ```

but the current version of Dispatch only accepts ASCII characters correctly.

```scala
property("POST non-ascii chars body and get response") = forAll(cyrillic) { (sample: String) =>
val res = Http(
localhost / "echobody" << sample > as.String
)
res() ?= ("POST" + sample)
}
```

The above test fails as:

```
[info] ! Basic.POST non-ascii chars body and get response: Falsified after 1 passed tests.
[info] > Labels of failing property:
[info] Expected "POSTҽ" but got "POST"
[info] > ARG_0: "ҽ"
```

This has surfaced as:
- [json string post encoding issue](https://groups.google.com/d/msg/dispatch-scala/3kcij5-15pY/LulUXnLemEgJ) posted by Young Gyu Park
- eed3si9n/scalaxb#231 Encoding problems (not using encoding from WSDL but defaults to dispatch's default?) reported by @hedefalk

As a related topic I should also link [ouch -- bitten by default (request) charset (_ == UTF-8)](https://groups.google.com/d/msg/dispatch-scala/0DMCsmLYCiY/mClejgg-QaAJ).

### what's going on?

> Charset is not its own header, but a parameter of content-type.
> http://www.w3.org/International/O-HTTP-charset

While this statement is correct, it does not address the encoding issue, since Async Http Client (AHC) has a specific property called `setBodyEncoding(String)` to control the encoding of body text, and does not seem look into `Content-Type` HTTP header. This is confirmed by the above test. So "body encoding" remains `null`, and [AHC seems to default on ASCII](https://github.com/AsyncHttpClient/async-http-client/blob/f4ad6ad876dffd7d05f7e9769b984c5d52d01333/providers/grizzly/src/main/java/org/asynchttpclient/providers/grizzly/bodyhandler/StringBodyHandler.java#L50-L52):

```java
if (charset == null) {
charset = Charsets.ASCII_CHARSET.name();
}
```

In other words, Dispatch is currently falling back on ASCII (or whatever Grizzly util's `Charsets.ASCII_CHARSET` resolves to) silently.

## what this pull req changes

First, I admit there's no clean solution to this situation. What's clear is that given no `charset` specification, the recipient of the message must interpret it as ISO-5589-1 (Latin-1). Does that logic apply to HTTP sending library? I don't think it does. If we do want to send anything other than Latin-1 encoded bytes, we have to put HTTP header.

So I added the following:

```scala
def setBodyEncoding(charset: String) =
subject.underlying { _.setBodyEncoding(charset) }
def setContentType(mediaType: String, charset: String) =
subject.underlying { _.setHeader("Content-Type", mediaType + "; charset=" + charset) }
```

It sucks that there are two methods for this, but one is Java encoding name the other is HTTP encoding name, so I kept it separate. Using the above methods, the users can set the `<<` encoding explicitly.

Next, I'm defaulting `<<(String)` to `text/plain; charset=UTF-8`:

```scala
/** Set request body to a given string,
* - set method to POST if currently GET,
* - set HTTP Content-Type to "text/plain; charset=UTF-8" if unspecified,
* - and set body encoding to UTF-8 if unspecified. */
def << (body: String) = {
defaultMethod("POST").defaultContentType("text/plain", "UTF-8").
defaultBodyEncoding("UTF-8").setBody(body)
}
```

Since `String` is Unicode string, it can carry characters well outside of the range of ASCII, so using UTF-8 I think is a safer fallback when the user simply passes in `<<("ҽ")` since it has a better chance of transmitting it to correctly.