The "somewhat racey" comment, you mean? :)

This comment is actually referring to the ability of this callback to keep doing stuff if taskConsumer.stop() is called after the callback has started running, since it runs in a thread we don't control.

*BUT* the issue you brought up is real as well. The challenge is that once the TaskRunner notifies the consumer that a task is done, we have some stuff we want to do:

- Remove task from the "currently running" list in the queue
- Insert new segments (if any)
- Create new subtasks (if any)
- Update status in DB

The first one "shouldn't" fail (modulo bugs). The others can all fail due to DB problems. Unless we take advantage of the fact that this stuff all happens in the same DB and use a single transaction (currently not possible due to the TaskStorage abstraction) I don't think it's possible to ensure we can be 100% consistent in terms of never having either one of:

- task marked successful in DB but commit callback failed
- task marked failed in DB but commit callback succeeded

If we care deeply about consistency we can go a different way, and consider either taking advantage of the fact that this stuff is all in the same DB *or* doing a commit process that looks like:

1- Update status in DB, and then try to create new segments and new subtasks, but don't be too concerned if the latter fails
2- Have something else that polls the DB and repairs consistency errors by creating "missing" segments and subtasks for committed statuses