enthought/distarray — pull request #174 by markkness (2014-03-01 00:15:14+00:00)
Title: Feature/purge command

This PR provides some functions to clean out various keys from all the engines.
It is still a little bit WIP, in particular I have seen the tests hang, but it should be ready for some discussion.

This extends the `Context` to provide two pieces of functionality:
1) It will keep track of the keys that it sends to all of the engines.
2) It has ability to query the engines to discover what keys are actually present there.

Using part 1, the `Context` can clean up the keys that it knowingly sent to the engines. The `cleanup_keys()` method does just this. One can also call the `delete_key(key)` method to delete a specific key. (The Distarray class now uses this to clean itself up in its `__del__` method.)

Ideally, this ought to remove all of the keys from the engines. But if not, there is also a `purge_keys()` method, part 2, that discovers all the keys and deletes them. This would provide more thorough cleanup.

There is also a combination `cleanup_keys_checked()` method that will clean up the first way, then clean up the second way, with the option to assert if there are extra keys still there. (The client test cases now use this.)

I envision a typical use case, as when a computation is done, the caller will clean up this way.

One complication is the `Context._comm_key` which persists with the context, and should not be removed by the cleanup methods, as this is needed as long as the context persists. This is handled somewhat awkwardly here. I have also had trouble ensuring that this is always cleaned up.

It would be nice to add a new `Makefile` target, e.g. `cleanup_cluster`, to call these methods on the cluster, so one could manually cause a cleanup step. This PR does not currently include such an option.