Repository: googleapis/google-cloud-python

Issue:
Port Datastore onto GAPIC
### What
Port the gRPC code path for Datastore onto GAPIC. The package is available [here](https://pypi.python.org/pypi/gapic-google-cloud-datastore-v1).

### Why
This is not a beta-blocker, since there already is gRPC code for Datastore, but it is preferred.


Discussion:
@bjwatson Two questions:
1. Why doesn't the new package depend on `google-gax>=0.15` (which has the new auth helpers)?
2. Are we in a situation where some of the GAPIC packages depend on `0.13 <= google-gax < 0.14` while others depend on `0.14 <= google-gax < 0.15` (as this one does)?

@dhermes Only because @geigerj says that he needs to make some small updates to the GAPIC generator for it to be compatible with `google-gax>=0.15`. He might be able to get that done today, and then we can build all packages with the newer `google-gax` dependency.

I thought I would get this in your hands right away, rather than waiting for the necessary fixes to support the newer auth helpers. However, if you think it's a more efficient use of your time to wait, then we can do that.

@bjwatson Sorry I didn't notice this until now, but the [current form](https://github.com/googleapis/api-client-staging/blob/f9305a6602771140c954939cf00b8ab09101d0c3/generated/python/grpc-google-cloud-datastore-v1/google/datastore/v1/datastore_pb2.py#L891-L895) of `grpc-google-cloud-datastore-v1` can't run without gRPC. This means we can't get access to the protobufs without forcing `grpc` to be installed.

Right now we support Protobuf-over-HTTP and Protobuf-over-gRPC, so we need access to the protobuf classes in use cases where gRPC isn't available (i.e. App Engine).

The generated protobuf classes currently live in [`datastore._generated`](https://github.com/GoogleCloudPlatform/google-cloud-python/tree/a10cf56445b51750c4d9bc1f3b5bde2522f92caf/datastore/google/cloud/datastore/_generated), which is how we import them. In order to support gRPC we have `datastore_pb2.py` and `datastore_grpc_pb2.py` in the `_generated` module

/cc @jonparrott 

@dhermes Does https://github.com/googleapis/api-client-staging/pull/106/commits/6649c45915223fa76fb390f9c12d90cb567a1496 do what you need? Do you also need changes to `setup.py`? I see that your `setup.py` [depends on `grpcio` regardless](https://github.com/GoogleCloudPlatform/google-cloud-python/blob/master/datastore/setup.py#L52-L55).

Note that this will become unnecessary once https://github.com/grpc/grpc/issues/8056 is released in `grpcio==1.1.0`.

Yes it does do what I had in mind.

We **had** conditional checks in our `setup.py`, but @jonparrott discouraged that since it confuses wheel building. But maybe I am wrong on how I remember that.

AFAIK, even though `grpcio` is in our `setup.py` for packages like `google-cloud-datastore`, it doesn't stop from installing into a `lib/` dir in an App Engine app. @jonparrott can confirm or deny or throw his hands up in sadness over packaging?
`grpcio` will indeed be vendored into `lib` for app engine and explode on import.
@bjwatson Ping me once released?
@bjwatson Is this a Datastore-specific hand-edit, or do we intend to provide these conditional imports across the gRPC packages we maintain? If it's the latter, we should figure out what the timeline for `grpcio 1.1.0` is and whether we need to automate this in our tooling.
@geigerj @bjwatson We would love to see a new release of `grpcio` because of https://github.com/grpc/grpc/issues/7849, so I am on team "pushing for release"
@dhermes We're going to wait on porting Datastore to GAPIC until after the beta announcement. We discussed and we don't want to take any chances on destabilizing the `google-cloud-python` code for the four APIs. The only one that will go beta with GAPIC is Logging.
SGTM
@lukesneeringer @dhermes If we're blocking the `x-goog-api-client` header updates on this issue, then it's `P0`.
@bjwatson Those are already in for HTTP. I can get them in to the gRPC transport before the process is finished (#3066 and #3064 will give you an idea of what the process looks like).
Ok, thanks Danny!
@dhermes This is now done, question mark?
Almost. One follow-up PR, coming in hot.

Resolving patch:
diff --git a/datastore/google/cloud/datastore/_gax.py b/datastore/google/cloud/datastore/_gax.py
--- a/datastore/google/cloud/datastore/_gax.py
+++ b/datastore/google/cloud/datastore/_gax.py
@@ -103,6 +103,64 @@ class GAPICDatastoreAPI(datastore_client.DatastoreClient):
     :param kwargs: Keyword arguments to pass to constructor.
     """
 
+    def lookup(self, *args, **kwargs):
+        """Perform a ``lookup`` request.
+
+        A light wrapper around the the base method from the parent class.
+        Intended to provide exception re-mapping (from GaxError to our
+        native errors).
+
+        :type args: tuple
+        :param args: Positional arguments to pass to base method.
+
+        :type kwargs: dict
+        :param kwargs: Keyword arguments to pass to base method.
+
+        :rtype: :class:`.datastore_pb2.LookupResponse`
+        :returns: The returned protobuf response object.
+        """
+        with _grpc_catch_rendezvous():
+            return super(GAPICDatastoreAPI, self).lookup(*args, **kwargs)
+
+    def run_query(self, *args, **kwargs):
+        """Perform a ``runQuery`` request.
+
+        A light wrapper around the the base method from the parent class.
+        Intended to provide exception re-mapping (from GaxError to our
+        native errors).
+
+        :type args: tuple
+        :param args: Positional arguments to pass to base method.
+
+        :type kwargs: dict
+        :param kwargs: Keyword arguments to pass to base method.
+
+        :rtype: :class:`.datastore_pb2.RunQueryResponse`
+        :returns: The returned protobuf response object.
+        """
+        with _grpc_catch_rendezvous():
+            return super(GAPICDatastoreAPI, self).run_query(*args, **kwargs)
+
+    def begin_transaction(self, *args, **kwargs):
+        """Perform a ``beginTransaction`` request.
+
+        A light wrapper around the the base method from the parent class.
+        Intended to provide exception re-mapping (from GaxError to our
+        native errors).
+
+        :type args: tuple
+        :param args: Positional arguments to pass to base method.
+
+        :type kwargs: dict
+        :param kwargs: Keyword arguments to pass to base method.
+
+        :rtype: :class:`.datastore_pb2.BeginTransactionResponse`
+        :returns: The returned protobuf response object.
+        """
+        with _grpc_catch_rendezvous():
+            return super(GAPICDatastoreAPI, self).begin_transaction(
+                *args, **kwargs)
+
     def commit(self, *args, **kwargs):
         """Perform a ``commit`` request.
 
@@ -122,6 +180,45 @@ def commit(self, *args, **kwargs):
         with _grpc_catch_rendezvous():
             return super(GAPICDatastoreAPI, self).commit(*args, **kwargs)
 
+    def rollback(self, *args, **kwargs):
+        """Perform a ``rollback`` request.
+
+        A light wrapper around the the base method from the parent class.
+        Intended to provide exception re-mapping (from GaxError to our
+        native errors).
+
+        :type args: tuple
+        :param args: Positional arguments to pass to base method.
+
+        :type kwargs: dict
+        :param kwargs: Keyword arguments to pass to base method.
+
+        :rtype: :class:`.datastore_pb2.RollbackResponse`
+        :returns: The returned protobuf response object.
+        """
+        with _grpc_catch_rendezvous():
+            return super(GAPICDatastoreAPI, self).rollback(*args, **kwargs)
+
+    def allocate_ids(self, *args, **kwargs):
+        """Perform an ``allocateIds`` request.
+
+        A light wrapper around the the base method from the parent class.
+        Intended to provide exception re-mapping (from GaxError to our
+        native errors).
+
+        :type args: tuple
+        :param args: Positional arguments to pass to base method.
+
+        :type kwargs: dict
+        :param kwargs: Keyword arguments to pass to base method.
+
+        :rtype: :class:`.datastore_pb2.AllocateIdsResponse`
+        :returns: The returned protobuf response object.
+        """
+        with _grpc_catch_rendezvous():
+            return super(GAPICDatastoreAPI, self).allocate_ids(
+                *args, **kwargs)
+
 
 def make_datastore_api(client):
     """Create an instance of the GAPIC Datastore API.


Test patch:
diff --git a/datastore/unit_tests/test__gax.py b/datastore/unit_tests/test__gax.py
--- a/datastore/unit_tests/test__gax.py
+++ b/datastore/unit_tests/test__gax.py
@@ -124,6 +124,69 @@ def _get_target_class():
     def _make_one(self, *args, **kwargs):
         return self._get_target_class()(*args, **kwargs)
 
+    def test_lookup(self):
+        from google.cloud.gapic.datastore.v1 import datastore_client
+
+        patch1 = mock.patch.object(
+            datastore_client.DatastoreClient, '__init__',
+            return_value=None)
+        patch2 = mock.patch.object(datastore_client.DatastoreClient, 'lookup')
+        patch3 = mock.patch(
+            'google.cloud.datastore._gax._grpc_catch_rendezvous')
+
+        with patch1 as mock_constructor:
+            ds_api = self._make_one()
+            mock_constructor.assert_called_once_with()
+            with patch2 as mock_lookup:
+                with patch3 as mock_catch_rendezvous:
+                    mock_catch_rendezvous.assert_not_called()
+                    ds_api.lookup(None, True, bb='cc')
+                    mock_lookup.assert_called_once_with(None, True, bb='cc')
+                    mock_catch_rendezvous.assert_called_once_with()
+
+    def test_run_query(self):
+        from google.cloud.gapic.datastore.v1 import datastore_client
+
+        patch1 = mock.patch.object(
+            datastore_client.DatastoreClient, '__init__',
+            return_value=None)
+        patch2 = mock.patch.object(
+            datastore_client.DatastoreClient, 'run_query')
+        patch3 = mock.patch(
+            'google.cloud.datastore._gax._grpc_catch_rendezvous')
+
+        with patch1 as mock_constructor:
+            ds_api = self._make_one()
+            mock_constructor.assert_called_once_with()
+            with patch2 as mock_run_query:
+                with patch3 as mock_catch_rendezvous:
+                    mock_catch_rendezvous.assert_not_called()
+                    ds_api.run_query('47a', none=None)
+                    mock_run_query.assert_called_once_with('47a', none=None)
+                    mock_catch_rendezvous.assert_called_once_with()
+
+    def test_begin_transaction(self):
+        from google.cloud.gapic.datastore.v1 import datastore_client
+
+        patch1 = mock.patch.object(
+            datastore_client.DatastoreClient, '__init__',
+            return_value=None)
+        patch2 = mock.patch.object(
+            datastore_client.DatastoreClient, 'begin_transaction')
+        patch3 = mock.patch(
+            'google.cloud.datastore._gax._grpc_catch_rendezvous')
+
+        with patch1 as mock_constructor:
+            ds_api = self._make_one()
+            mock_constructor.assert_called_once_with()
+            with patch2 as mock_begin_transaction:
+                with patch3 as mock_catch_rendezvous:
+                    mock_catch_rendezvous.assert_not_called()
+                    ds_api.begin_transaction('a', 'b', [], key='kei')
+                    mock_begin_transaction.assert_called_once_with(
+                        'a', 'b', [], key='kei')
+                    mock_catch_rendezvous.assert_called_once_with()
+
     def test_commit(self):
         from google.cloud.gapic.datastore.v1 import datastore_client
 
@@ -144,6 +207,50 @@ def test_commit(self):
                     mock_commit.assert_called_once_with(1, 2, a=3)
                     mock_catch_rendezvous.assert_called_once_with()
 
+    def test_rollback(self):
+        from google.cloud.gapic.datastore.v1 import datastore_client
+
+        patch1 = mock.patch.object(
+            datastore_client.DatastoreClient, '__init__',
+            return_value=None)
+        patch2 = mock.patch.object(
+            datastore_client.DatastoreClient, 'rollback')
+        patch3 = mock.patch(
+            'google.cloud.datastore._gax._grpc_catch_rendezvous')
+
+        with patch1 as mock_constructor:
+            ds_api = self._make_one()
+            mock_constructor.assert_called_once_with()
+            with patch2 as mock_rollback:
+                with patch3 as mock_catch_rendezvous:
+                    mock_catch_rendezvous.assert_not_called()
+                    ds_api.rollback(11, 12, arp='marp')
+                    mock_rollback.assert_called_once_with(11, 12, arp='marp')
+                    mock_catch_rendezvous.assert_called_once_with()
+
+    def test_allocate_ids(self):
+        from google.cloud.gapic.datastore.v1 import datastore_client
+
+        patch1 = mock.patch.object(
+            datastore_client.DatastoreClient, '__init__',
+            return_value=None)
+        patch2 = mock.patch.object(
+            datastore_client.DatastoreClient, 'allocate_ids')
+        patch3 = mock.patch(
+            'google.cloud.datastore._gax._grpc_catch_rendezvous')
+
+        with patch1 as mock_constructor:
+            ds_api = self._make_one()
+            mock_constructor.assert_called_once_with()
+            with patch2 as mock_allocate_ids:
+                with patch3 as mock_catch_rendezvous:
+                    mock_catch_rendezvous.assert_not_called()
+                    ds_api.allocate_ids(
+                        'hey', 'bai', bye=(47, 4), shy={'a': 4})
+                    mock_allocate_ids.assert_called_once_with(
+                        'hey', 'bai', bye=(47, 4), shy={'a': 4})
+                    mock_catch_rendezvous.assert_called_once_with()
+
 
 @unittest.skipUnless(_HAVE_GRPC, 'No gRPC')
 class Test_make_datastore_api(unittest.TestCase):