[Discuss] Persistent Connections

_Our current priority is chameleon, I'm just dumping my thoughts here and we can focus on this later._

**Brief**: This is about reducing the chatter between consumer apps using galaxy-client and galaxy.

## The Problem
> Ruby's own net/http uses a little known behavior where by default an "Connection: close" header is appended to each request

_source: http://www.igvita.com/2011/10/04/optimizing-http-keep-alive-and-pipelining/_

If you are not so fresh on HTTP 1.1 Keepalive, read the article.

Galaxy-client uses ActiveResource which uses net/http to make the all api calls galaxy. ActiveResource does not support persistent HTTP connections and sends a Connection: close after each request it makes.

There may be opportunity here to reduce the latency of consumer app transactions, and strengthen our ability to scale, by reusing TCP connections to galaxy.

## The Solution
Patch it! Then test and see the difference, re-evaluate the problem.

Here are two `tcpdump` dumps the network chatter from chameleon to galaxy, during the reload of the home page (Regions#show)
* without keepalive: 119 packets sent http://paste/9dq 
* with keepalive: 32 packets sent http://paste/03g

_notice the Connection: Close/keealive headers_

## Implementation
There are two ways we could go about implementing this, using the [activeresource-persistent](https://github.com/railsware/activeresource-persistent/) gem, or writing it ourselves. Both approaches are relatively lightweight.

### activeresource-persistent
Easy to get going, does not seem to ever close the connection, it lets the keepalive timeout run out
Makes use of [net-http-persistent](https://github.com/drbrain/net-http-persistent), which looks like a healthy gem, some traction, regular commits. Supports SSL reuse (? don't know much about that, do we use SSL for consumer to galaxy communication?)

### a custom patch written by us
Simple to implement, greater control, 
Ability to control when connection close is sent. (Any significant advantage to closing the connection after, say every consumer app client request, rather than letting the server timeout the connection?)
Allows us to consider [alternate](http://bibwild.wordpress.com/2012/04/30/ruby-http-performance-shootout-redux/) HTTP client libs. Like Curb, which is already a dependency of both chameleon and galaxy.

## Testing
If we get an ActiveResource patch for connection reuse, we can deploy a test environment with a consumer app and a galaxy on separate hosts, and profile with/without the patch. James has so prior to this sort of stuff.

## Production
On the other end, our production nginx reverse-proxies support HTTP keepalive connections. The [keepalive timeout](http://nginx.org/en/docs/http/ngx_http_core_module.html#keepalive_timeout) will need to be set carefully, probably to a low value.
TCP connections would be saved across HTTP api calls over the the consumer app to nginx leg. Unicorn does not keepalive by design, so this does not apply there.

I would suggest deploying this to chameleon only initially, using it as a spearhead for the idea. We can quickly see if it causes any trouble and that way we are not breaking existing expectations (unlike existing consumer apps)

## Open questions
* Do we like this?
* Does it make a difference? (need perf env test)
* Which patch do we want?
* Any significant advantage to closing the connection manually?
* Do we want this only on chameleon only to begin with?

The persistent-connections branch is a patch to galaxy-client with activeresource-persistent for your convenience.