cqlengine/cqlengine — pull request #178 by dvdotsenko (2014-04-14 05:18:02+00:00)
Title: adding callbacks support to BatchQuery

In order to allow secondary tasks to be chained to the end of batch, BatchQuery instances allow callbacks to be registered with the batch, to be executed immediately after the batch executes.

Let's say an iterative action over a group of records needs to be followed up by a "maintenance" task like "Update count of children on parent model." If your were to kick off that task following manipulation on each child, while the batch is not executed, the reads the maintenance task would do would operate on old data since the batch is not yet commited. (You also risk kicking off too many of the same tasks, while only one is needed at the end of the batch.)

The callbacks attached to a given batch instance are executed only if the batch executes. If the batch is used as a context manager and an exception bubbles up, the queued up callbacks will not be run.

The callback should be a callable object that accepts at least one (positional) argument - instance of the batch object.

```
def example_callback_handler(batch, *args, **kwargs):
pass
```

Multiple callbacks can be attached to same BatchQuery instance both, at the point of instantiation and later
directly to the batch object instance.

The callbacks collection is implemented as a ``set``, which naturally deduplicates the callback handlers:

```
# let's add a few of the same callbacks at init
batch = BatchQuery(callbacks=([
example_callback_handler,
example_callback_handler
])

# and one more time same callback now using the collection API on the batch instance
batch.callbacks.add(example_callback_handler)

assert len(batch.callbacks) == 1
assert batch.callbacks == {example_callback_handler}

# this, of course, also means that if you remove a callback,
# you negate "all additions" of that callback
batch.callbacks.remove(example_callback_handler)

assert len(batch.callbacks) == 0
```

In order to benefit from the natural deduplication of the callback handlers it is allowed to pass in tuples of
callback and its positional arguments. This allows one to pre-package additional call arguments for the callable without the need to wrap the callable and args into a closure or ``functools.partial``.

```
batch = BatchQuery(callbacks=([
(example_callback_handler, 'some', 'args'),
(example_callback_handler, 'some', 'args'),
(example_callback_handler, 'some other', 'args'),
])

batch.callbacks.add((example_callback_handler, 'some other', 'args'))

assert len(batch.callbacks) == 2
assert batch.callbacks == {
(example_callback_handler, 'some', 'args'),
(example_callback_handler, 'some other', 'args')
}
```

Note that the tuple callback object format prevents the possibily of essentially same callback firing multiple times, which might happen if you package same callback and arguments several times with ``functools.partial``, which returns a new callable with each call.

Arguments packaged into callbacks communicated as tuples are passed to callback handler **after** the ``batch`` argument.

```
call_args = []

def callback_handler(batch, *args, **kwargs):
call_args.append([batch] + list(args))

batch = BatchQuery(callbacks=([(callback_handler, 'some', 'args')])

batch.execute()

assert call_args == [[batch, 'some', 'args']]
```

Failure in any of the callbacks does not affect the batch's execution, as the callbacks are started after the execution of the batch is complete and no effort to "roll it back" is made.