Repository: googleapis/google-cloud-python

Issue:
Cloud Spanner Client should prevent nested transactions
Cloud Spanner does not support [nested transactions](https://en.wikipedia.org/wiki/Nested_transaction). But the client library does allow users to call `database.run_in_transaction` inside of the callback that is provided to an outer `database.run_in_transaction`. This is misleading since users might believe that this will behave like a nested transaction but in reality these will be two independent transactions. This is confusing and a source of bugs. Specifically the inner transaction might succeed while the outer might ABORT, in which case the callback will be rerun thus rerunning the inner transaction. 
We should prevent this from happening by detecting and raising an error if someone calls `database.run_in_transaction` inside the callback provided to `database.run_in_transaction`.  Note that this needs to be done before Beta since this is a known breaking change.
Also note that if we come across a use case where we do want to allow calling a nested `database.run_in_transaction`, we should be able to enable that in future without making a breaking change. 

cc @jgeewax @bjwatson 


Discussion:
@tseaver This can wait until you have time to work on the Spanner integration tests.
Note that this also is beta blocking.
Thanks for the reminder, @vkedia. I'm reviewing the Python Spanner issues, and adding the `status: release blocking` to any that are Beta blocking. That way we can see a [hotlist](https://github.com/GoogleCloudPlatform/google-cloud-python/issues?utf8=%E2%9C%93&q=is%3Aopen%20label%3A%22status%3A%20release%20blocking%22%20label%3A%22api%3A%20spanner%22) for remaining issue.
Can this be reopened? I don't think the fix actually fixes this.
Reopened on behalf of @vkedia, so the remaining issues can be discussed further.
I reviewed the code; it does exactly what it says on the tin. What is the problem exactly?
The exact requirements are if someone calls `database.run_in_transaction` inside the callback passed to `database.run_in_transaction` it should blow up. I don't think that would happen with the fix. You can verify that by writing a test that does this. Infact such a test should have been there in the PR in the first place.
There is [a test that does this](https://github.com/tseaver/google-cloud-python/blob/36cb9341fc0e521e2822b9770de0e380445e9bc0/spanner/tests/unit/test_transaction.py#L50).
It is not the same thing. That test is calling `session.transaction`. I am talking about `database.run_in_transaction`.  The issue is when you call `database.run_in_transaction` again, it will checkout a new session so it will keep on working.
Right way to fix this would be somehow track in a thread local context or something analogous that there is an ongoing transaction in this thread and use that to prevent any other transaction from being opened.
@tseaver I think I agree with Vikas here. The session checkout would still fundamentally allow...

```python
def bad_news():
    database.run_in_transaction(unit_of_work)

database.run_in_transaction(bad_news)
```

~Note: This is so incredibly unusual in Python that I am struggling to figure out how to properly prevent it.~ Nevermind, I think I know what to do.
@lukesneeringer FWIW, the callback passed to `Database.run_in_transaction` *ought* to be using only the locals it is passed.
@tseaver Agreed, but we have no way to enforce that.

In fairness, relying on this kind of weird scoping is uncommon in Python, so this entire issue is well into the edge case territory ("if someone aims a gun at their foot...").

Regardless, I will have a PR in a moment.
Calling `database.run_in_transaction` from within the unit-of-work callback (e.g., `bad_news`) will check out a new session, start a new transaction in it, and call the passed other-unit-of-work callback.  I don't see how we can prevent that.

Resolving patch:
diff --git a/spanner/google/cloud/spanner/database.py b/spanner/google/cloud/spanner/database.py
--- a/spanner/google/cloud/spanner/database.py
+++ b/spanner/google/cloud/spanner/database.py
@@ -15,6 +15,7 @@
 """User friendly container for Cloud Spanner Database."""
 
 import re
+import threading
 
 import google.auth.credentials
 from google.gax.errors import GaxError
@@ -79,6 +80,7 @@ def __init__(self, database_id, instance, ddl_statements=(), pool=None):
         self.database_id = database_id
         self._instance = instance
         self._ddl_statements = _check_ddl_statements(ddl_statements)
+        self._local = threading.local()
 
         if pool is None:
             pool = BurstyPool()
@@ -332,8 +334,20 @@ def run_in_transaction(self, func, *args, **kw):
         :rtype: :class:`datetime.datetime`
         :returns: timestamp of committed transaction
         """
-        with SessionCheckout(self._pool) as session:
-            return session.run_in_transaction(func, *args, **kw)
+        # Sanity check: Is there a transaction already running?
+        # If there is, then raise a red flag. Otherwise, mark that this one
+        # is running.
+        if getattr(self._local, 'transaction_running', False):
+            raise RuntimeError('Spanner does not support nested transactions.')
+        self._local.transaction_running = True
+
+        # Check out a session and run the function in a transaction; once
+        # done, flip the sanity check bit back.
+        try:
+            with SessionCheckout(self._pool) as session:
+                return session.run_in_transaction(func, *args, **kw)
+        finally:
+            self._local.transaction_running = False
 
     def batch(self):
         """Return an object which wraps a batch.


Test patch:
diff --git a/spanner/tests/unit/test_database.py b/spanner/tests/unit/test_database.py
--- a/spanner/tests/unit/test_database.py
+++ b/spanner/tests/unit/test_database.py
@@ -223,7 +223,7 @@ def __init__(self, scopes=(), source=None):
                 self._scopes = scopes
                 self._source = source
 
-            def requires_scopes(self): # pragma: NO COVER
+            def requires_scopes(self):  # pragma: NO COVER
                 return True
 
             def with_scopes(self, scopes):
@@ -663,6 +663,29 @@ def test_run_in_transaction_w_args(self):
         self.assertEqual(session._retried,
                          (_unit_of_work, (SINCE,), {'until': UNTIL}))
 
+    def test_run_in_transaction_nested(self):
+        from datetime import datetime
+
+        # Perform the various setup tasks.
+        instance = _Instance(self.INSTANCE_NAME, client=_Client())
+        pool = _Pool()
+        session = _Session(run_transaction_function=True)
+        session._committed = datetime.now()
+        pool.put(session)
+        database = self._make_one(self.DATABASE_ID, instance, pool=pool)
+
+        # Define the inner function.
+        inner = mock.Mock(spec=())
+
+        # Define the nested transaction.
+        def nested_unit_of_work():
+            return database.run_in_transaction(inner)
+
+        # Attempting to run this transaction should raise RuntimeError.
+        with self.assertRaises(RuntimeError):
+            database.run_in_transaction(nested_unit_of_work)
+        self.assertEqual(inner.call_count, 0)
+
     def test_batch(self):
         from google.cloud.spanner.database import BatchCheckout
 
@@ -900,11 +923,15 @@ class _Session(object):
 
     _rows = ()
 
-    def __init__(self, database=None, name=_BaseTest.SESSION_NAME):
+    def __init__(self, database=None, name=_BaseTest.SESSION_NAME,
+                 run_transaction_function=False):
         self._database = database
         self.name = name
+        self._run_transaction_function = run_transaction_function
 
     def run_in_transaction(self, func, *args, **kw):
+        if self._run_transaction_function:
+            func(*args, **kw)
         self._retried = (func, args, kw)
         return self._committed