Project: datamanagement
Issue DM-1177: Implement sharable just-in-time subchunk management on the worker
The current subchunk management on the worker is very simple. When a query fragment is selected for execution, it builds exactly the subchunks it needs, performs the queries, and destroys the subchunks. This presents a problem when we have concurrent queries that need the same subchunks. Thus an earlier query will destroy its subchunks (some/all of which are needed by another concurrent query) and cause the later queries to fail. The suggested approach is as follows: * Each query, instead of building subchunks on its own, delegates responsibility for subchunk creation/deletion to another class (e.g. SubChunkManager, [but try to find a better name]). * Each query requests a subchunks and releases subchunks using a lock()/unlock() mechanism (in the SubChunkManager API), or wrapped in an interface like boost::lock_guard wraps boost::mutex where acquisition and deletion can be handled by scoping. (It might even be possible to use boost::lock_guard mechanism.) * Now SubChunkManager blocks calls to lock() until it has created the necessary subchunks and modified its records to note that these subchunks are needed by one additional query). A subsequent unlock() call from the client indicates that SCM can decrement its counter and destroy the appropriate subchunks. I (danielw) think the data structure to manage this is straightforward and the code for creation/deletion of subchunks already exists, but it might not be obvious how to route the plumbing so that queries can request/release appropriately, to an instance (probably only one per worker instance) owned by an object somewhere else. Please feel free to use a better or more maintainable approach.
Story points: 15