MIST-190 - Agent guest info, stream, async queues and Guest Job Log

A set of valid actions has been defined. Actions have a type (info, stream, or async) and a set of stages that act as a pipeline template. Only the stages are exposed via the configuration file. Stages are configured with service name, method, args via the same file. When an action request is received, a pipeline is generated from the action's stages and appropriate request/response structs. The pipeline is then sent to the guest runner for processing.

Each guest is provided a guest runner containing three queues: info, stream, async. The first two are simply throttle queues, holding empty structs that behave as locks. Actions using those are executed in real time and the result is returned to the user. The third, async, is a queue that holds pipelines themselves. Once queued, a response is sent back to the user immediately and execution takes place in the background. If a stage in a pipeline fails, it will bail out rather than continue.

Async jobs have their status monitored and preserved in a job log containing the last 100 actions. It is persisted via kvite. A pipeline id is provided via the X-Guest-Job-ID header for later status checking via /guests/{guestid}/jobs/{jobid}. Retrieving the last X actions is done via /guests/{guestid}/jobs(?limit=[0-9]+). The log provides action name, queue time, start time, current status (queued, running, errored, complete), and a message (e.g. an error message).

On deletion, the runner is destroyed and the job log is deleted.
Hmm. You're right, there's no incremental flushing of the responsewriter. ReverseProxy.copyResponse (of course, not exported) allows flushing on time intervals. Something along the same line as http://stackoverflow.com/questions/19292113/not-buffered-http-responsewritter-in-golang might work, manually flushing after writing a small buffer. I'll dig around a bit more.