Support ad-hoc chef runs

Biggest complaint: It's synchronous. This is going to re-introduce long-running http connections.