Repository: googleapis/google-cloud-python

Issue:
Batch dataset.put and dataset.delete.
Like dataset.get, there should be a dataset.delete and dataset.put that accepts a list of keys / entities.



Discussion:
I need to get on #336 too and hopefully these can come together. We should actually hash out the discussion in #188 as well, since many of our core patterns are very JavaScript-like and may not be the right way forward.

Given a list of entities in hand, what is the advantage of a "convenience" method on the dataset to save or delete them, against just iterating the list and calling `save` or `delete` on each one?  The `get_entity` and
`get_entities` methods on a dataset are there for when you don't already have the entities.

I guess there might be some utility to a `delete` method taking only keys, so that the user doesn't need to either fetch them via `get_entities` or convert them to protobufs to hand to the connection's `delete_entities` method.

I think `dataset.delete` and `dataset.put` are valid, somewhat like `ndb.put_multi` and `ndb.delete_multi`.

The Zen of Python says:

> There should be one– and preferably only one –obvious way to do it.

What use is `Dataset.put()`?  To make it work you have to have the entity objects in hand already, so why not just:

```
for entity in entities_to_put:
    entity.save()
```

If `Dataset.delete()` takes entities, the same applies (entities already have a `delete()` method).  If it takes keys (only), then it might be useful, if you can reasonably expect that there is a way to have a bunch of keys in hand without also having the corresponding entities.

It's mostly for performances reason. Doing:

```
for entity in entities_to_put:
    entity.save()
```

Means one RPC for each `save`. Batching means one RPC for all saves. (and same for deletes).

Isn't that what you get from using a transaction?  All the adds / saves / deletes get sent as a single RPC.

That's meant for a transactional context, where if one fails, they all fail.

I think it makes sense to have this be a mode/flag/subclass of `Dataset` rather than extra methods. 1. it keeps the interface of the class cleaner, 2. it adds support for batch mutations which contain a mix of mutation types. Ideally syntax could be similar to transaction syntax. 

```
with dataset.batch_mutation() as b:
    for entity in entities_to_put:
        b.put(entity)
    for key in keys_to_delete:
        b.delete(key)
```

IMO we need to focus on feature parity with the API before we can add nice sugar like that.

As @silvolu says, we need methods for batch puts and deletes.

What is the expectation for error propagation after a non-transactional batch update / delete, where some of the operations failed?

@tseaver There is none. If guarantees need to be made about mutation success transactions must be used. 

@dhermes It's not necessarily sugar. If instead of immediately calling `connection.commit()` dataset writes to a mutation_pb buffer and immediately flushes that buffer (by calling `connection.commit()`) then a batch_mutator class could override that behavior and only flush the buffer on context exit. I think there is a lot of user experience value in having the same method signatures in a different context for batch mutations. 

This implementation also works quite nicely for transactions. I have a class hierarchy I made for the `Entity`less implementation of the library which could easily be adapted, if you would be interested. 

@elibixby are you saying there is no way to detect the failures in batch mode?  Who would use an API like that, and why would we go out of our way to support them?

@tseaver this is fundamentally a property of the datastore, not the API. You could for instance make a bunch of batch writes, and then do other stuff. Then when you need to use the data, in a transaction retrieve all of the writes you made, redo the writes that failed, and operate on your data as normal. This will be faster than simply making all the writes in the transaction, as fewer writes will have to propagate through the datastore, (since transactions demand snapshot isolation and strong consistency)

EDIT: to expand on this, transactions are for making mutations that are dependent upon a certain state in the datastore (e.g. updating data based on it's current value)

OTOH you may want to make a huge number of mutations that don't depend on the state of the datastore. If you want to add 10,000 entities to the datastore (during a short time, and not dependent on the current content of the datastore), it would be EXTREMELY bad to slam the datastore with 10,000 api calls. This is a frequent use case during datastore initialization or clean up (in testing for example)

@dhermes we have examples of context-manager-based batching in Storage already:

```
@property
def batch(self):
    """Return a context manager which defers/batches updates.

    E.g., to batch multiple updates to a bucket::

        >>> with bucket.batch:
        ...     bucket.enable_versioning()
        ...     bucket.disable_website()

    or for a key::

        >>> with key.batch:
        ...     key.content_type = 'image/jpeg'
        ...     key.content_encoding = 'gzip'

    Updates will be aggregated and sent as a single call to
    :meth:`_patch_properties` IFF the ``with`` block exits without
    an exception.

    :rtype: :class:`_PropertyBatch`
    """
    return _PropertyBatch(self)
```

@tseaver with the major refactorings, is this still under the radar?

It's right on the horizon (for one of us at least).


Resolving patch:
diff --git a/gcloud/datastore/batch.py b/gcloud/datastore/batch.py
new file mode 100644
--- /dev/null
+++ b/gcloud/datastore/batch.py
@@ -0,0 +1,169 @@
+# Copyright 2014 Google Inc. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Create / interact with a batch of updates / deletes."""
+
+from gcloud.datastore import _implicit_environ
+from gcloud.datastore import datastore_v1_pb2 as datastore_pb
+
+
+class Batch(object):
+    """An abstraction representing a collected group of updates / deletes.
+
+    Used to build up a bulk mutuation.
+
+    For example, the following snippet of code will put the two ``save``
+    operations and the delete operatiuon into the same mutation, and send
+    them to the server in a single API request::
+
+      >>> from gcloud import datastore
+      >>> batch = Batch()
+      >>> batch.put(entity1)
+      >>> batch.put(entity2)
+      >>> batch.delete(key3)
+      >>> batch.commit()
+
+    You can also use a batch as a context manager, in which case the
+    ``commit`` will be called automatically if its block exits without
+    raising an exception::
+
+      >>> with Batch() as batch:
+      ...     batch.put(entity1)
+      ...     batch.put(entity2)
+      ...     batch.delete(key3)
+
+    By default, no updates will be sent if the block exits with an error::
+
+      >>> from gcloud import datastore
+      >>> dataset = datastore.get_dataset('dataset-id')
+      >>> with Batch as batch:
+      ...   do_some_work(batch)
+      ...   raise Exception() # rolls back
+    """
+
+    def __init__(self, dataset_id=None, connection=None):
+        """ Construct a batch.
+
+        :type dataset_id: :class:`str`.
+        :param dataset_id: The ID of the dataset.
+
+        :type connection: :class:`gcloud.datastore.connection.Connection`
+        :param connection: The connection used to connect to datastore.
+
+        :raises: :class:`ValueError` if either a connection or dataset ID
+                 are not set.
+        """
+        self._connection = connection or _implicit_environ.CONNECTION
+        self._dataset_id = dataset_id or _implicit_environ.DATASET_ID
+
+        if self._connection is None or self._dataset_id is None:
+            raise ValueError('A batch must have a connection and '
+                             'a dataset ID set.')
+
+        self._mutation = datastore_pb.Mutation()
+
+    @property
+    def dataset_id(self):
+        """Getter for dataset ID in which the batch will run.
+
+        :rtype: :class:`str`
+        :returns: The dataset ID in which the batch will run.
+        """
+        return self._dataset_id
+
+    @property
+    def connection(self):
+        """Getter for connection over which the batch will run.
+
+        :rtype: :class:`gcloud.datastore.connection.Connection`
+        :returns: The connection over which the batch will run.
+        """
+        return self._connection
+
+    @property
+    def mutation(self):
+        """Getter for the current mutation.
+
+        Every batch is committed with a single Mutation
+        representing the 'work' to be done as part of the batch.
+        Inside a batch, calling ``batch.put()`` with an entity, or
+        ``batch.delete`` with a key, builds up the mutation.
+        This getter returns the Mutation protobuf that
+        has been built-up so far.
+
+        :rtype: :class:`gcloud.datastore.datastore_v1_pb2.Mutation`
+        :returns: The Mutation protobuf to be sent in the commit request.
+        """
+        return self._mutation
+
+    def put(self, entity):
+        """Remember an entity's state to be saved during ``commit``.
+
+        .. note::
+           Any existing properties for the entity will be replaced by those
+           currently set on this instance.  Already-stored properties which do
+           not correspond to keys set on this instance will be removed from
+           the datastore.
+
+        .. note::
+           Property values which are "text" ('unicode' in Python2, 'str' in
+           Python3) map to 'string_value' in the datastore;  values which are
+           "bytes" ('str' in Python2, 'bytes' in Python3) map to 'blob_value'.
+
+        :type entity: :class:`gcloud.datastore.entity.Entity`
+        :param entity: the entity to be saved.
+
+        :raises: ValueError if entity has no key assigned.
+        """
+        if entity.key is None:
+            raise ValueError("Entity must have a key")
+
+        key_pb = entity.key.to_protobuf()
+        properties = dict(entity)
+        exclude = tuple(entity.exclude_from_indexes)
+
+        self.connection.save_entity(
+            self.dataset_id, key_pb, properties,
+            exclude_from_indexes=exclude, mutation=self.mutation)
+
+    def delete(self, key):
+        """Remember a key to be deleted durring ``commit``.
+
+        :type key: :class:`gcloud.datastore.key.Key`
+        :param key: the key to be deleted.
+
+        :raises: ValueError if key is not complete.
+        """
+        if key.is_partial:
+            raise ValueError("Key must be complete")
+
+        key_pb = key.to_protobuf()
+        self.connection.delete_entities(
+            self.dataset_id, [key_pb], mutation=self.mutation)
+
+    def commit(self):
+        """Commits the batch.
+
+        This is called automatically upon exiting a with statement,
+        however it can be called explicitly if you don't want to use a
+        context manager.
+        """
+        self.connection.commit(self._dataset_id, self.mutation)
+
+    def __enter__(self):
+        return self
+
+    def __exit__(self, exc_type, exc_val, exc_tb):
+        if exc_type is None:
+            self.commit()
diff --git a/gcloud/datastore/connection.py b/gcloud/datastore/connection.py
--- a/gcloud/datastore/connection.py
+++ b/gcloud/datastore/connection.py
@@ -421,7 +421,7 @@ def allocate_ids(self, dataset_id, key_pbs):
         return list(response.key)
 
     def save_entity(self, dataset_id, key_pb, properties,
-                    exclude_from_indexes=()):
+                    exclude_from_indexes=(), mutation=None):
         """Save an entity to the Cloud Datastore with the provided properties.
 
         .. note::
@@ -441,13 +441,24 @@ def save_entity(self, dataset_id, key_pb, properties,
         :type exclude_from_indexes: sequence of string
         :param exclude_from_indexes: Names of properties *not* to be indexed.
 
+        :type mutation: :class:`gcloud.datastore.datastore_v1_pb2.Mutation`
+                        or None.
+        :param mutation: If passed, the mutation protobuf into which the
+                         entity will be saved.  If None, use th result
+                         of calling ``self.mutation()``
+
         :rtype: tuple
         :returns: The pair (``assigned``, ``new_id``) where ``assigned`` is a
                   boolean indicating if a new ID has been assigned and
                   ``new_id`` is either ``None`` or an integer that has been
                   assigned.
         """
-        mutation = self.mutation()
+        if mutation is not None:
+            in_batch = True
+        else:
+            in_batch = False
+            mutation = self.mutation()
+
         key_pb = helpers._prepare_key_for_request(key_pb)
 
         # If the Key is complete, we should upsert
@@ -479,7 +490,7 @@ def save_entity(self, dataset_id, key_pb, properties,
 
         # If this is in a transaction, we should just return True. The
         # transaction will handle assigning any keys as necessary.
-        if self.transaction():
+        if in_batch or self.transaction():
             return False, None
 
         result = self.commit(dataset_id, mutation)
@@ -493,7 +504,7 @@ def save_entity(self, dataset_id, key_pb, properties,
 
         return False, None
 
-    def delete_entities(self, dataset_id, key_pbs):
+    def delete_entities(self, dataset_id, key_pbs, mutation=None):
         """Delete keys from a dataset in the Cloud Datastore.
 
         This method deals only with
@@ -508,13 +519,24 @@ def delete_entities(self, dataset_id, key_pbs):
         :type key_pbs: list of :class:`gcloud.datastore.datastore_v1_pb2.Key`
         :param key_pbs: The keys to delete from the datastore.
 
+        :type mutation: :class:`gcloud.datastore.datastore_v1_pb2.Mutation`
+                        or None.
+        :param mutation: If passed, the mutation protobuf into which the
+                         deletion will be saved.  If None, use th result
+                         of calling ``self.mutation()``
+
         :rtype: boolean
         :returns: ``True``
         """
-        mutation = self.mutation()
+        if mutation is not None:
+            in_batch = True
+        else:
+            in_batch = False
+            mutation = self.mutation()
+
         helpers._add_keys_to_request(mutation.delete, key_pbs)
 
-        if not self.transaction():
+        if not in_batch and not self.transaction():
             self.commit(dataset_id, mutation)
 
         return True


Test patch:
diff --git a/gcloud/datastore/test_batch.py b/gcloud/datastore/test_batch.py
new file mode 100644
--- /dev/null
+++ b/gcloud/datastore/test_batch.py
@@ -0,0 +1,203 @@
+# Copyright 2014 Google Inc. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import unittest2
+
+
+class TestBatch(unittest2.TestCase):
+
+    def _getTargetClass(self):
+        from gcloud.datastore.batch import Batch
+
+        return Batch
+
+    def _makeOne(self, dataset_id=None, connection=None):
+        return self._getTargetClass()(dataset_id=dataset_id,
+                                      connection=connection)
+
+    def test_ctor_missing_required(self):
+        from gcloud._testing import _Monkey
+        from gcloud.datastore import _implicit_environ
+
+        with _Monkey(_implicit_environ,
+                     DATASET_ID=None,
+                     CONNECTION=None):
+            self.assertRaises(ValueError, self._makeOne)
+            self.assertRaises(ValueError, self._makeOne, dataset_id=object())
+            self.assertRaises(ValueError, self._makeOne, connection=object())
+
+    def test_ctor_explicit(self):
+        from gcloud.datastore.datastore_v1_pb2 import Mutation
+        _DATASET = 'DATASET'
+        connection = _Connection()
+        batch = self._makeOne(dataset_id=_DATASET, connection=connection)
+
+        self.assertEqual(batch.dataset_id, _DATASET)
+        self.assertEqual(batch.connection, connection)
+        self.assertTrue(isinstance(batch.mutation, Mutation))
+
+    def test_ctor_implicit(self):
+        from gcloud._testing import _Monkey
+        from gcloud.datastore import _implicit_environ
+        from gcloud.datastore.datastore_v1_pb2 import Mutation
+        DATASET_ID = 'DATASET'
+        CONNECTION = _Connection()
+
+        with _Monkey(_implicit_environ,
+                     DATASET_ID=DATASET_ID,
+                     CONNECTION=CONNECTION):
+            batch = self._makeOne()
+
+        self.assertEqual(batch.dataset_id, DATASET_ID)
+        self.assertEqual(batch.connection, CONNECTION)
+        self.assertTrue(isinstance(batch.mutation, Mutation))
+
+    def test_put_entity_wo_key(self):
+        _DATASET = 'DATASET'
+        connection = _Connection()
+        batch = self._makeOne(dataset_id=_DATASET, connection=connection)
+
+        self.assertRaises(ValueError, batch.put, _Entity())
+
+    def test_put_entity_w_key(self):
+        _DATASET = 'DATASET'
+        _PROPERTIES = {'foo': 'bar'}
+        connection = _Connection()
+        batch = self._makeOne(dataset_id=_DATASET, connection=connection)
+        entity = _Entity(_PROPERTIES)
+        key = entity.key = _Key(_DATASET)
+
+        batch.put(entity)
+
+        self.assertEqual(
+            connection._saved,
+            (_DATASET, key._key, _PROPERTIES, (), batch.mutation))
+
+    def test_delete_w_partial_key(self):
+        _DATASET = 'DATASET'
+        connection = _Connection()
+        batch = self._makeOne(dataset_id=_DATASET, connection=connection)
+        key = _Key(_DATASET)
+        key._partial = True
+
+        self.assertRaises(ValueError, batch.delete, key)
+
+    def test_delete_w_completed_key(self):
+        _DATASET = 'DATASET'
+        connection = _Connection()
+        batch = self._makeOne(dataset_id=_DATASET, connection=connection)
+        key = _Key(_DATASET)
+
+        batch.delete(key)
+
+        self.assertEqual(
+            connection._deleted,
+            (_DATASET, [key._key], batch.mutation))
+
+    def test_commit(self):
+        _DATASET = 'DATASET'
+        connection = _Connection()
+        batch = self._makeOne(dataset_id=_DATASET, connection=connection)
+
+        batch.commit()
+
+        self.assertEqual(connection._committed, (_DATASET, batch.mutation))
+
+    def test_as_context_mgr_wo_error(self):
+        _DATASET = 'DATASET'
+        _PROPERTIES = {'foo': 'bar'}
+        connection = _Connection()
+        entity = _Entity(_PROPERTIES)
+        key = entity.key = _Key(_DATASET)
+
+        with self._makeOne(dataset_id=_DATASET,
+                           connection=connection) as batch:
+            batch.put(entity)
+
+        self.assertEqual(
+            connection._saved,
+            (_DATASET, key._key, _PROPERTIES, (), batch.mutation))
+        self.assertEqual(connection._committed, (_DATASET, batch.mutation))
+
+    def test_as_context_mgr_w_error(self):
+        _DATASET = 'DATASET'
+        _PROPERTIES = {'foo': 'bar'}
+        connection = _Connection()
+        entity = _Entity(_PROPERTIES)
+        key = entity.key = _Key(_DATASET)
+
+        try:
+            with self._makeOne(dataset_id=_DATASET,
+                               connection=connection) as batch:
+                batch.put(entity)
+                raise ValueError("testing")
+        except ValueError:
+            pass
+
+        self.assertEqual(
+            connection._saved,
+            (_DATASET, key._key, _PROPERTIES, (), batch.mutation))
+        self.assertEqual(connection._committed, None)
+
+
+class _CommitResult(object):
+
+    def __init__(self, *new_keys):
+        self.insert_auto_id_key = new_keys
+
+
+class _Connection(object):
+    _marker = object()
+    _committed = _saved = _deleted = None
+    _save_result = (False, None)
+
+    def __init__(self):
+        self._commit_result = _CommitResult()
+
+    def save_entity(self, dataset_id, key_pb, properties,
+                    exclude_from_indexes=(), mutation=None):
+        self._saved = (dataset_id, key_pb, properties,
+                       tuple(exclude_from_indexes), mutation)
+        return self._save_result
+
+    def delete_entities(self, dataset_id, key_pbs, mutation=None):
+        self._deleted = (dataset_id, key_pbs, mutation)
+
+    def commit(self, dataset_id, mutation):
+        self._committed = (dataset_id, mutation)
+        return self._commit_result
+
+
+class _Entity(dict):
+    key = None
+    exclude_from_indexes = ()
+
+
+class _Key(object):
+    _MARKER = object()
+    _key = 'KEY'
+    _partial = False
+    _path = None
+    _id = None
+    _stored = None
+
+    def __init__(self, dataset_id):
+        self.dataset_id = dataset_id
+
+    @property
+    def is_partial(self):
+        return self._partial
+
+    def to_protobuf(self):
+        return self._key
diff --git a/gcloud/datastore/test_connection.py b/gcloud/datastore/test_connection.py
--- a/gcloud/datastore/test_connection.py
+++ b/gcloud/datastore/test_connection.py
@@ -1075,6 +1075,25 @@ class Xact(object):
         mutation = conn.mutation()
         self.assertEqual(len(mutation.upsert), 1)
 
+    def test_save_entity_w_mutation_passed(self):
+        from gcloud.datastore import datastore_v1_pb2 as datastore_pb
+        from gcloud.datastore.entity import Entity
+        DATASET_ID = 'DATASET'
+        nested = Entity()
+        nested['bar'] = u'Bar'
+        key_pb = self._make_key_pb(DATASET_ID)
+        rsp_pb = datastore_pb.CommitResponse()
+        conn = self._makeOne()
+        http = conn._http = Http({'status': '200'}, rsp_pb.SerializeToString())
+        mutation = datastore_pb.Mutation()
+        result = conn.save_entity(DATASET_ID, key_pb, {'foo': nested},
+                                  mutation=mutation)
+        self.assertEqual(result, (False, None))
+        self.assertEqual(http._called_with, None)
+        conn_mutation = conn.mutation()
+        self.assertEqual(len(mutation.upsert), 1)
+        self.assertEqual(len(conn_mutation.upsert), 0)
+
     def test_delete_entities_wo_transaction(self):
         from gcloud.datastore import datastore_v1_pb2 as datastore_pb
 
@@ -1126,6 +1145,21 @@ class Xact(object):
         mutation = conn.mutation()
         self.assertEqual(len(mutation.delete), 1)
 
+    def test_delete_entities_w_mutation_passed(self):
+        from gcloud.datastore import datastore_v1_pb2 as datastore_pb
+        DATASET_ID = 'DATASET'
+        key_pb = self._make_key_pb(DATASET_ID)
+        rsp_pb = datastore_pb.CommitResponse()
+        conn = self._makeOne()
+        http = conn._http = Http({'status': '200'}, rsp_pb.SerializeToString())
+        mutation = datastore_pb.Mutation()
+        result = conn.delete_entities(DATASET_ID, [key_pb], mutation=mutation)
+        self.assertEqual(result, True)
+        self.assertEqual(http._called_with, None)
+        conn_mutation = conn.mutation()
+        self.assertEqual(len(mutation.delete), 1)
+        self.assertEqual(len(conn_mutation.delete), 0)
+
 
 class Http(object):