Repository: googleapis/google-cloud-python

Issue:
Datastore transactions do not work without context manager
[The docs](https://googlecloudplatform.github.io/gcloud-python/stable/datastore-transactions.html#gcloud.datastore.transaction.Transaction) state that you can use transactions without a context manager, but this is not exactly true because you need to resort to some private methods on `Client`:
- [this file](https://gist.github.com/Bogdanp/a86192d1a2a37dd48df5ebdbb39a49d1#file-context-py-L20) uses the context manager and produces correct output
- [this file](https://gist.github.com/Bogdanp/a86192d1a2a37dd48df5ebdbb39a49d1#file-failing-py-L20) uses transactions as described by the aforementioned docs and fails the assertion
- [this file](https://gist.github.com/Bogdanp/a86192d1a2a37dd48df5ebdbb39a49d1#file-passing-py-L20) uses transactions along with calls to `client._push_batch` and `client._pop_batch` and produces correct output

The issue is evident in [`Client.get_multi`](https://github.com/GoogleCloudPlatform/gcloud-python/blob/master/gcloud/datastore/client.py#L284) and [`Client.put_multi`](https://github.com/GoogleCloudPlatform/gcloud-python/blob/master/gcloud/datastore/client.py#L336) where `transaction` and `current` are `None` unless a `Batch` or `Transaction` has been pushed to the `_batch_stack` and the only time that happens is when you [`__enter__`](https://github.com/GoogleCloudPlatform/gcloud-python/blob/master/gcloud/datastore/batch.py#L273) a `Batch`.



Discussion:
@Bogdanp thanks for finding this!

In the case where the assertions fail, is it both values or just one of them that isn't getting set when you `t.commit()`?

Both values are updated. However, due to the fact that there is no [transaction id](https://github.com/GoogleCloudPlatform/gcloud-python/blob/master/gcloud/datastore/client.py#L292) "in scope" for those get and put requests, the serializability of the operations is not enforced.

Here's some sample output:

``` fish
(tempenv-3357153745070) ~ [1]> env DATASTORE_DATASET="test" DATASTORE_PROJECT_ID="test" DATASTORE_HOST="http://127.0.0.1:9999" python failing_test.py 
60
40
Traceback (most recent call last):
  File "failing_test.py", line 54, in <module>
    assert a1["balance"] == 100 - n * 10
AssertionError
```

I just added a couple of print statements before the assertions in the failing example:

``` python
    print a1["balance"]
    print a2["balance"]

    assert a1["balance"] == 100 - n * 10
    assert a2["balance"] == n * 10
```

`Client.put_multi` is never called in your failing example:  the changes are all accumulated inside the transaction's `_commit_request` protobuf.  I think the error is that the transaction ID ([fetched during `Transaction.begin`](https://github.com/GoogleCloudPlatform/gcloud-python/blob/835d2dc97e03941bc2004b6e819220a21f613c99/gcloud/datastore/transaction.py#L130)) is not being stored into that protobuf.

I just tried to write a system test replicating your failing report, and found that it passes:

``` python
    def test_transaction_via_imperative_begin_commit(self):
        BEFORE_1 = 100
        BEFORE_2 = 0
        TRANSFER_AMOUNT = 40
        key1 = Config.CLIENT.key('account', '123')
        account1 = datastore.Entity(key=key1)
        account1['balance'] = BEFORE_1
        key2 = Config.CLIENT.key('account', '234')
        account2 = datastore.Entity(key=key2)
        account2['balance'] = BEFORE_2
        Config.CLIENT.put_multi([account1, account2])

        def transfer_funds(client, from_key, to_key, amount):
            xact = client.transaction()
            xact.begin()
            from_account = client.get(from_key)
            to_account = client.get(to_key)
            from_account['balance'] -= amount
            to_account['balance'] += amount

            xact.put(from_account)
            xact.put(to_account)
            xact.commit()

        transfer_funds(Config.CLIENT, key1, key2, TRANSFER_AMOUNT)

        after1 = Config.CLIENT.get(key1)
        after2 = Config.CLIENT.get(key2)
        self.assertEqual(after1['balance'], BEFORE_1 - TRANSFER_AMOUNT)
        self.assertEqual(after2['balance'], BEFORE_2 + TRANSFER_AMOUNT)
```

One issue might be that`Client.get_multi` is not guaranteed to return the items in the order of the requested keys:  can you update your failing example to compensate (e.g., either use `Client.get` or else check the keys of the returned entities)?

(Of course, my example is not threaded with multiple writers, so it is not really a good enough replica).

[Here](https://gist.github.com/Bogdanp/60fbf9655258b969c526a401a33dbfbc)'s a failing example w/o any `get_multi` calls.

``` fish
(tempenv-3357153745070) ~ [1]> env DATASTORE_DATASET="test" DATASTORE_PROJECT_ID="test" DATASTORE_HOST="http://127.0.0.1:9999" python failing_test.py 
60
40
Traceback (most recent call last):
  File "failing_test.py", line 56, in <module>
    assert a1["balance"] == 100 - n * 10
AssertionError
```

BTW, I picked this example specifically because it's what's being used in the [official docs](https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/datastore/api/snippets.py#L649) (note the use of `get_multi`).

Are you sure [this](https://github.com/GoogleCloudPlatform/gcloud-python/blob/master/gcloud/datastore/client.py#L292) isn't the culprit? `transaction_id` ends up being `None` there unless `_push_batch` is called prior to `get`/`get_multi`.

I was still investigating:  you are correct that `get`/`get_multi` do not provide any support for doing the reads in the context of an _explicit_ transaction.  I have a PR in progress which will add that support.


Resolving patch:
diff --git a/gcloud/datastore/client.py b/gcloud/datastore/client.py
--- a/gcloud/datastore/client.py
+++ b/gcloud/datastore/client.py
@@ -223,7 +223,7 @@ def current_transaction(self):
         if isinstance(transaction, Transaction):
             return transaction
 
-    def get(self, key, missing=None, deferred=None):
+    def get(self, key, missing=None, deferred=None, transaction=None):
         """Retrieve an entity from a single key (if it exists).
 
         .. note::
@@ -244,15 +244,19 @@ def get(self, key, missing=None, deferred=None):
         :param deferred: (Optional) If a list is passed, the keys returned
                          by the backend as "deferred" will be copied into it.
 
+        :type transaction: :class:`gcloud.datastore.transaction.Transaction`
+        :param transaction: (Optional) Transaction to use for read consistency.
+                            If not passed, uses current transaction, if set.
+
         :rtype: :class:`gcloud.datastore.entity.Entity` or ``NoneType``
         :returns: The requested entity if it exists.
         """
         entities = self.get_multi(keys=[key], missing=missing,
-                                  deferred=deferred)
+                                  deferred=deferred, transaction=transaction)
         if entities:
             return entities[0]
 
-    def get_multi(self, keys, missing=None, deferred=None):
+    def get_multi(self, keys, missing=None, deferred=None, transaction=None):
         """Retrieve entities, along with their attributes.
 
         :type keys: list of :class:`gcloud.datastore.key.Key`
@@ -268,6 +272,10 @@ def get_multi(self, keys, missing=None, deferred=None):
                          by the backend as "deferred" will be copied into it.
                          If the list is not empty, an error will occur.
 
+        :type transaction: :class:`gcloud.datastore.transaction.Transaction`
+        :param transaction: (Optional) Transaction to use for read consistency.
+                            If not passed, uses current transaction, if set.
+
         :rtype: list of :class:`gcloud.datastore.entity.Entity`
         :returns: The requested entities.
         :raises: :class:`ValueError` if one or more of ``keys`` has a project
@@ -281,7 +289,8 @@ def get_multi(self, keys, missing=None, deferred=None):
             if current_id != self.project:
                 raise ValueError('Keys do not match project')
 
-        transaction = self.current_transaction
+        if transaction is None:
+            transaction = self.current_transaction
 
         entity_pbs = _extended_lookup(
             connection=self.connection,


Test patch:
diff --git a/gcloud/datastore/test_client.py b/gcloud/datastore/test_client.py
--- a/gcloud/datastore/test_client.py
+++ b/gcloud/datastore/test_client.py
@@ -231,8 +231,10 @@ def _get_multi(*args, **kw):
         self.assertEqual(_called_with[0][1]['keys'], [key])
         self.assertTrue(_called_with[0][1]['missing'] is None)
         self.assertTrue(_called_with[0][1]['deferred'] is None)
+        self.assertTrue(_called_with[0][1]['transaction'] is None)
 
     def test_get_hit(self):
+        TXN_ID = '123'
         _called_with = []
         _entity = object()
 
@@ -246,12 +248,13 @@ def _get_multi(*args, **kw):
 
         key, missing, deferred = object(), [], []
 
-        self.assertTrue(client.get(key, missing, deferred) is _entity)
+        self.assertTrue(client.get(key, missing, deferred, TXN_ID) is _entity)
 
         self.assertEqual(_called_with[0][0], ())
         self.assertEqual(_called_with[0][1]['keys'], [key])
         self.assertTrue(_called_with[0][1]['missing'] is missing)
         self.assertTrue(_called_with[0][1]['deferred'] is deferred)
+        self.assertEqual(_called_with[0][1]['transaction'], TXN_ID)
 
     def test_get_multi_no_keys(self):
         creds = object()
@@ -412,6 +415,40 @@ def test_get_multi_hit(self):
         self.assertEqual(list(result), ['foo'])
         self.assertEqual(result['foo'], 'Foo')
 
+    def test_get_multi_hit_w_transaction(self):
+        from gcloud.datastore.key import Key
+
+        TXN_ID = '123'
+        KIND = 'Kind'
+        ID = 1234
+        PATH = [{'kind': KIND, 'id': ID}]
+
+        # Make a found entity pb to be returned from mock backend.
+        entity_pb = _make_entity_pb(self.PROJECT, KIND, ID, 'foo', 'Foo')
+
+        # Make a connection to return the entity pb.
+        creds = object()
+        client = self._makeOne(credentials=creds)
+        client.connection._add_lookup_result([entity_pb])
+
+        key = Key(KIND, ID, project=self.PROJECT)
+        txn = client.transaction()
+        txn._id = TXN_ID
+        result, = client.get_multi([key], transaction=txn)
+        new_key = result.key
+
+        # Check the returned value is as expected.
+        self.assertFalse(new_key is key)
+        self.assertEqual(new_key.project, self.PROJECT)
+        self.assertEqual(new_key.path, PATH)
+        self.assertEqual(list(result), ['foo'])
+        self.assertEqual(result['foo'], 'Foo')
+
+        cw = client.connection._lookup_cw
+        self.assertEqual(len(cw), 1)
+        _, _, _, transaction_id = cw[0]
+        self.assertEqual(transaction_id, TXN_ID)
+
     def test_get_multi_hit_multiple_keys_same_project(self):
         from gcloud.datastore.key import Key
 
diff --git a/system_tests/datastore.py b/system_tests/datastore.py
--- a/system_tests/datastore.py
+++ b/system_tests/datastore.py
@@ -417,7 +417,7 @@ def test_query_distinct_on(self):
 
 class TestDatastoreTransaction(TestDatastore):
 
-    def test_transaction(self):
+    def test_transaction_via_with_statement(self):
         entity = datastore.Entity(key=Config.CLIENT.key('Company', 'Google'))
         entity['url'] = u'www.google.com'
 
@@ -432,6 +432,40 @@ def test_transaction(self):
         self.case_entities_to_delete.append(retrieved_entity)
         self.assertEqual(retrieved_entity, entity)
 
+    def test_transaction_via_explicit_begin_get_commit(self):
+        # See https://github.com/GoogleCloudPlatform/gcloud-python/issues/1859
+        # Note that this example lacks the threading which provokes the race
+        # condition in that issue:  we are basically just exercising the
+        # "explict" path for using transactions.
+        BEFORE_1 = 100
+        BEFORE_2 = 0
+        TRANSFER_AMOUNT = 40
+        key1 = Config.CLIENT.key('account', '123')
+        account1 = datastore.Entity(key=key1)
+        account1['balance'] = BEFORE_1
+        key2 = Config.CLIENT.key('account', '234')
+        account2 = datastore.Entity(key=key2)
+        account2['balance'] = BEFORE_2
+        Config.CLIENT.put_multi([account1, account2])
+        self.case_entities_to_delete.append(account1)
+        self.case_entities_to_delete.append(account2)
+
+        xact = Config.CLIENT.transaction()
+        xact.begin()
+        from_account = Config.CLIENT.get(key1, transaction=xact)
+        to_account = Config.CLIENT.get(key2, transaction=xact)
+        from_account['balance'] -= TRANSFER_AMOUNT
+        to_account['balance'] += TRANSFER_AMOUNT
+
+        xact.put(from_account)
+        xact.put(to_account)
+        xact.commit()
+
+        after1 = Config.CLIENT.get(key1)
+        after2 = Config.CLIENT.get(key2)
+        self.assertEqual(after1['balance'], BEFORE_1 - TRANSFER_AMOUNT)
+        self.assertEqual(after2['balance'], BEFORE_2 + TRANSFER_AMOUNT)
+
     def test_failure_with_contention(self):
         contention_prop_name = 'baz'
         local_client = clone_client(Config.CLIENT)