flux-framework/flux-core — pull request #123 by garlick (2014-12-19 23:02:12+00:00)
Title: implement matchtag protocol enhancement

This change allows clients to assign a unique integer 'matchtag' to a request which will be echoed back in the corresponding response. This makes it easier for a client to manage multiple outstanding requests.

The `flux_t` handle implements a "tag pool". Unique matchtags can be obtained using `flux_matchtag_alloc()` and retired using `flux_matchtag_free()`.

`flux_json_rpc()` hides tag management from callers.

`flux_response_recvmsg()` now has a `matchtag` argument and will manage the requeuing of messages that come in but do not match the requested tag. Alternatively, calling it with a matchtag of 0 disables tag matching.

`flux_json_request()` also has a `matchtag` argument.

The previous method of matching on topic strings limited concurrency since at most one request per topic string could be outstanding. This was especially problematic in situations where a request generated an unknown number of replies, such as with `kvs_watch()`.

The integer matchtag scheme was borrowed from the [9P protocol](http://man.cat-v.org/inferno/5/intro) in Plan 9. One difference is 9P uses a 16-bit tag value while I went with an 8-bit one. Easy to change that later if we need more concurrency, but 255 outstanding requests per handle should get us pretty far for now and allows a trivial tagpool implementation.