it might be slightly more efficient if we refactor this to accept a list of partition index so we don't need to create too many future objects and thus not too many threads waiting for the future to finish ...