tent/tentd — pull request #73 by jonasschneider (2013-01-13 18:52:10+00:00)
Title: Streaming API discussion

The initial points outlined by @titanous:

- [ ] The API should be read-only. Pipelined/keep-alive HTTP POSTs are fine for post creation.
- [ ] We need to keep shared hosting providers in mind. This means that clients should support a 30x redirect to a different server.
- [ ] The streaming API will not be an optional part of the protocol, so there's no need to specify the endpoint in the profile.
- [ ] We should support both length-prefixed JSON as well as JSON in Server-Sent Events (http://dev.w3.org/html5/eventsource/).
- [ ] The URL should be the same (/posts) with a different Accept header (text/event-stream for SSE, we need to find one for length-prefixed JSON)
- [ ] The endpoint should support all of the filtering parameters that the current /posts endpoint does.
- [ ] I don't want to switch to a framework like Goliath that depends on EventMachine. While EM is useful in some situations, I'm not a fan of using it for tentd.
- [ ] Sinatra supports streaming out of the box on most Rack servers, we should look into how that works.
- [ ] To keep deployment simplicity, we should not require spinning up a separate process to do streaming. We can support an optional separate process deployment, but this must not be required.
- [ ] We need to sort out advisory events/backpressure for clients that do not read quickly enough (we should not be buffering posts infinitely for slow clients).

This is the first draft of what I came up with. Check it out and run

curl -H "Accept: application/vnd.tent.v0+length-prefixed-json" http://localhost:5000/posts

The concurrency is utterly broken and completely open to DoS and probably countless other attacks, and the performance sucks. Yesyes. I'm open to everything.

Things still to be done:
- [ ] extract serialization into `TentD::Serializer` or similar
- [ ] use an external thread pool library, or manage the streaming threads manually in a sensible way (i.e. limit to N per app/IP or similar, clean up occasionally)
- [ ] use a shared DB connection and process-internal broadcast
- [ ] check if that JSON format is right
- [ ] add SSE format
- [ ] clean up the raw HTTP fail
- [ ] maybe consolidate the `Post.fetch_with_permissions`/`fetch_all` API

How do you feel about this?