Repository: googleapis/google-cloud-python

Issue:
Datastore Key's reference is missing location identifier in app argument in to_legacy_urlsafe()
@dhermes I put this as a new issue because I thought people might have a problem with this too. Please feel free to edit/remove this issue if it is not appropriate to put it here.

I was running `pip install --upgrade google-cloud-datastore` to get latest version that has the `key.to_legacy_urlsafe()` implemented in [#3491](https://github.com/GoogleCloudPlatform/google-cloud-python/pull/3491). I tested the method and I found out that the urlsafe generated from the method is still not the same as datastore's urlsafe.

I have implemented my own function to generate the urlsafe key before the update was released and so far it worked properly for me (I can share my code if anybody wants). So I compared my function with the to_legacy_urlsafe and I found the problem:


```
reference = _app_engine_key_pb2.Reference(
        app=self.project,
        path=_to_legacy_path(self._path),  # Avoid the copy.
        name_space=self.namespace,
        )
```

In the app argument, the location identifier is missing unlike in my function where I concatenated the "s~" at the beginning of the string. I think this is the cause of the problem why it was not getting the urlsafe correct, although I don't understand why it needs the location identifier in front of the project name in the first place.

Do you think I should send a PR to fix this?


Discussion:
@michaelenglo I'll let @jonparrott handle this. I voiced this concern when writing the original PR and he said we shouldn't worry about it.

I would suggest that you rely on `from_legacy_urlsafe` much more than `to_legacy_urlsafe`, i.e. these are really meant as a way to go **FROM** legacy **TO** "modern", not the other direction.
As for sending a PR, there is no "right" way to fix this. The `app` may begin with `dev~`, `s~`, `e~` and possibly other strings I am unaware of. There is no way to determine which of these is appropriate based on the project string.
Because there's no "right" way to fix this, I think we should just document that the string returned by our `to_legacy_urlsafe` is equivalent, but not identical to the key returned by `ndb`.
I am a bit concerned about this because urlsafe play an important role in our app. Currently in the implementation of our Document Control app, every document file has its information (in datastore), and its blob (in cloud storage). The way we name each document is by its urlsafe key (since we thought it's always unique, we decided to use it instead of creating app-specific identifier). Now, If we cannot guarantee that both services in Std and Flex generate the same urlsafe key consistently, then there is a chance that in the future we can have corrupted files lost in the database.

Is using urlsafe as unique name for objects something that we should avoid? If yes, then what is the proper way for mapping entities to blob objects in cloud storage?

And I wonder why if NDB has a method to create a ursafe key that is identical from the one in Datastore, what prevents the Client Library from having one?
@michaelenglo You can get around this by just using the same identifier here that is using in App Engine standard. It's fairly like your application uses `s~` as the project ID. You can deserialize a protobuf from a `urlsafe` value to find out.
> And I wonder why if NDB has a method to create a ursafe key that is identical from the one in Datastore, what prevents the Client Library from having one?

`ndb` runs in App Engine, so it has access to the application ID **according to App Engine**, which contains the prefix (`s~`, `e~`, etc.). If you manually add the correct prefix, you'll get the same values (see above).
Hey @michaelenglo,
I just wanted to follow up on this item. I think we are probably leaning toward leaving this as-is (except maybe adding some documentation), but I wanted to give you an opportunity to follow up on it first.
@michaelenglo And if you want truly identical behavior in your application, you should determine which prefix your app uses (it's almost certainly `s~`) and then

> manually add the correct prefix

via `datastore.Client(project='s~your-app-id')`

Resolving patch:
diff --git a/datastore/google/cloud/datastore/key.py b/datastore/google/cloud/datastore/key.py
--- a/datastore/google/cloud/datastore/key.py
+++ b/datastore/google/cloud/datastore/key.py
@@ -298,7 +298,7 @@ def to_protobuf(self):
 
         return key
 
-    def to_legacy_urlsafe(self):
+    def to_legacy_urlsafe(self, location_prefix=None):
         """Convert to a base64 encode urlsafe string for App Engine.
 
         This is intended to work with the "legacy" representation of a
@@ -310,13 +310,26 @@ def to_legacy_urlsafe(self):
         .. note::
 
             The string returned by ``to_legacy_urlsafe`` is equivalent, but
-            not identical, to the string returned by ``ndb``.
+            not identical, to the string returned by ``ndb``. The location
+            prefix may need to be specified to obtain identical urlsafe
+            keys.
+
+        :type location_prefix: str
+        :param location_prefix: The location prefix of an App Engine project
+                                ID. Often this value is 's~', but may also be
+                                'e~', or other location prefixes currently
+                                unknown.
 
         :rtype: bytes
         :returns: A bytestring containing the key encoded as URL-safe base64.
         """
+        if location_prefix is None:
+            project_id = self.project
+        else:
+            project_id = location_prefix + self.project
+
         reference = _app_engine_key_pb2.Reference(
-            app=self.project,
+            app=project_id,
             path=_to_legacy_path(self._path),  # Avoid the copy.
             name_space=self.namespace,
         )


Test patch:
diff --git a/datastore/tests/unit/test_key.py b/datastore/tests/unit/test_key.py
--- a/datastore/tests/unit/test_key.py
+++ b/datastore/tests/unit/test_key.py
@@ -35,6 +35,9 @@ class TestKey(unittest.TestCase):
     _URLSAFE_EXAMPLE2 = b'agZzfmZpcmVyDwsSBEtpbmQiBVRoaW5nDA'
     _URLSAFE_APP2 = 's~fire'
     _URLSAFE_FLAT_PATH2 = ('Kind', 'Thing')
+    _URLSAFE_EXAMPLE3 = b'ahhzfnNhbXBsZS1hcHAtbm8tbG9jYXRpb25yCgsSBFpvcnAYWAw'
+    _URLSAFE_APP3 = 'sample-app-no-location'
+    _URLSAFE_FLAT_PATH3 = ('Zorp', 88)
 
     @staticmethod
     def _get_target_class():
@@ -408,6 +411,13 @@ def test_to_legacy_urlsafe_strip_padding(self):
         # Make sure it started with base64 padding.
         self.assertNotEqual(len(self._URLSAFE_EXAMPLE2) % 4, 0)
 
+    def test_to_legacy_urlsafe_with_location_prefix(self):
+        key = self._make_one(
+            *self._URLSAFE_FLAT_PATH3,
+            project=self._URLSAFE_APP3)
+        urlsafe = key.to_legacy_urlsafe(location_prefix='s~')
+        self.assertEqual(urlsafe, self._URLSAFE_EXAMPLE3)
+
     def test_from_legacy_urlsafe(self):
         klass = self._get_target_class()
         key = klass.from_legacy_urlsafe(self._URLSAFE_EXAMPLE1)
@@ -430,6 +440,15 @@ def test_from_legacy_urlsafe_needs_padding(self):
         self.assertIsNone(key.namespace)
         self.assertEqual(key.flat_path, self._URLSAFE_FLAT_PATH2)
 
+    def test_from_legacy_urlsafe_with_location_prefix(self):
+        klass = self._get_target_class()
+        # Make sure it will have base64 padding added.
+        key = klass.from_legacy_urlsafe(self._URLSAFE_EXAMPLE3)
+
+        self.assertEqual(key.project, self._URLSAFE_APP3)
+        self.assertIsNone(key.namespace)
+        self.assertEqual(key.flat_path, self._URLSAFE_FLAT_PATH3)
+
     def test_is_partial_no_name_or_id(self):
         key = self._make_one('KIND', project=self._DEFAULT_PROJECT)
         self.assertTrue(key.is_partial)