Repository: googleapis/google-cloud-python

Issue:
datastore: Nested entity parsing fails
https://cloud.google.com/appengine/docs/python/ndb/properties#structured

In order to write entities with structured properties, I tried to read them first, to figure out the format.

First `StructuredProperty`.

``` Python
class Item_Meta(ndb.Model):
    xy = ndb.IntegerProperty(repeated = True)

class Item(ndb.Model):
    meta = ndb.StructuredProperty(Item_Meta)

item = Item()
item.meta = Item_Meta(xy = [100, 200])
item.put() # 5710239819104256
```

``` Python
>>> datastore.api.get([datastore.Key('Item', 5710239819104256)])
[<Entity[{'kind': u'Item', 'id': 5710239819104256L}] {u'meta.xy': [100L, 200L]}>]
```

Next `LocalStructuredProperty`.

> Although a `StructuredProperty` can be repeated and a `StructuredProperty` can contain another `StructuredProperty`, beware: if one structured property contains another, only one of them can be repeated. A work-around is to use `LocalStructuredProperty`, which does not have this constraint.

``` Python
class Item_Meta(ndb.Model):
    xy = ndb.IntegerProperty(repeated = True)

class Item(ndb.Model):
    meta = ndb.LocalStructuredProperty(Item_Meta, repeated = True)

item = Item()
item.meta = [Item_Meta(xy = [100, 200])]
item.put() # 6217263929622528
```

``` Python
>>> datastore.api.get([datastore.Key('Item', 6217263929622528)])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/user/.local/lib/python2.7/site-packages/gcloud/datastore/api.py", line 229, in get
    entities.append(helpers.entity_from_protobuf(entity_pb))
  File "/home/user/.local/lib/python2.7/site-packages/gcloud/datastore/helpers.py", line 53, in entity_from_protobuf
    value = _get_value_from_property_pb(property_pb)
  File "/home/user/.local/lib/python2.7/site-packages/gcloud/datastore/helpers.py", line 237, in _get_value_from_property_pb
    return _get_value_from_value_pb(property_pb.value)
  File "/home/user/.local/lib/python2.7/site-packages/gcloud/datastore/helpers.py", line 217, in _get_value_from_value_pb
    result = [_get_value_from_value_pb(x) for x in value_pb.list_value]
  File "/home/user/.local/lib/python2.7/site-packages/gcloud/datastore/helpers.py", line 214, in _get_value_from_value_pb
    result = entity_from_protobuf(value_pb.entity_value)
  File "/home/user/.local/lib/python2.7/site-packages/gcloud/datastore/helpers.py", line 48, in entity_from_protobuf
    key = key_from_protobuf(pb.key)
  File "/home/user/.local/lib/python2.7/site-packages/gcloud/datastore/helpers.py", line 105, in key_from_protobuf
    return Key(*path_args, namespace=namespace, dataset_id=dataset_id)
  File "/home/user/.local/lib/python2.7/site-packages/gcloud/datastore/key.py", line 76, in __init__
    self._path = self._combine_args()
  File "/home/user/.local/lib/python2.7/site-packages/gcloud/datastore/key.py", line 133, in _combine_args
    child_path = self._parse_path(self._flat_path)
  File "/home/user/.local/lib/python2.7/site-packages/gcloud/datastore/key.py", line 94, in _parse_path
    raise ValueError('Key path must not be empty.')
ValueError: Key path must not be empty.
```



Discussion:
@pdknsk Are you sure that's the correct stacktrace?

Also, we don't intend to support `LocalStructruredProperty` in `gcloud.datastore` but are hoping to put `ndb` in `gcloud.datastore` at some point. (See #557.)

I expect the request to return a `bytes` object of the local structured property serialized via `pickle`.

Yes, it's correct.

According to `ndb` [source](https://chromium.googlesource.com/external/googleappengine/python/+/7e0ab775c587657f0f93a3134f2db99e46bb98bd/google/appengine/ext/ndb/model.py#185) the value of `LocalStructuredProperty` uses _the standard "protocol buffer" encoding_.

1. I didn't read closely enough, the stacktrace is from parsing the response.
2. After posting the `pickle` comment I immediately wondering if the protocol buffer was in fact the underlying format. My bad.

I will try to spin up an App Engine app to test this, but in the mean time you could run this code (at 0.4.0) and post your findings

``` python
from gcloud.datastore import _implicit_environ
from gcloud import datastore

datastore.set_defaults()

key_pb = datastore.Key('Item', 6217263929622528).to_protobuf()

cnxn = _implicit_environ.CONNECTION
dataset_id = _implicit_environ.DATASET_ID

results, missing, deferred = cnxn.lookup(dataset_id, [key_pbs])

print 'Results:'
for r in results:
    print r

print 'Missing:'
for m in missing:
    print m

print 'Deferred:'
for d in deferred:
    print d
```

Then the protobufs in `results` have `__repr__` defined so they print quite nicely.

Looks good, I think.

``` Python
Results:
key {
  partition_id {
    dataset_id: "s~boxheart-net"
  }
  path_element {
    kind: "Item"
    id: 5707702298738688
  }
}
property {
  name: "meta"
  value {
    list_value {
      entity_value {
        property {
          name: "xy"
          value {
            list_value {
              integer_value: 100
            }
            list_value {
              integer_value: 200
            }
          }
        }
      }
      indexed: false
    }
  }
}

Missing:
Deferred:
```

Debugging now, recreated the entity in https://gist.github.com/dhermes/94746659780f25e362d4

@pdknsk Can you confirm something for me?

```
# Same code as above
# ...
results, missing, deferred = cnxn.lookup(dataset_id, [key_pbs])

entity_pb = results[0]
print entity_pb.property[0].value.list_value[0].entity_value.HasField('key')
```

I'm fairly certain this will be `False` and we can avoid a thorny fix.

Thanks a ton for finding and reporting this!

PS I renamed the issue to describe the more generic issue at hand.


Resolving patch:
diff --git a/gcloud/datastore/helpers.py b/gcloud/datastore/helpers.py
--- a/gcloud/datastore/helpers.py
+++ b/gcloud/datastore/helpers.py
@@ -45,7 +45,10 @@ def entity_from_protobuf(pb):
     :rtype: :class:`gcloud.datastore.entity.Entity`
     :returns: The entity derived from the protobuf.
     """
-    key = key_from_protobuf(pb.key)
+    key = None
+    if pb.HasField('key'):
+        key = key_from_protobuf(pb.key)
+
     entity_props = {}
     exclude_from_indexes = []
 


Test patch:
diff --git a/gcloud/datastore/test_helpers.py b/gcloud/datastore/test_helpers.py
--- a/gcloud/datastore/test_helpers.py
+++ b/gcloud/datastore/test_helpers.py
@@ -104,6 +104,48 @@ def test_mismatched_value_indexed(self):
         with self.assertRaises(ValueError):
             self._callFUT(entity_pb)
 
+    def test_entity_no_key(self):
+        from gcloud.datastore import _datastore_v1_pb2 as datastore_pb
+
+        entity_pb = datastore_pb.Entity()
+        entity = self._callFUT(entity_pb)
+
+        self.assertEqual(entity.key, None)
+        self.assertEqual(dict(entity), {})
+
+    def test_nested_entity_no_key(self):
+        from gcloud.datastore import _datastore_v1_pb2 as datastore_pb
+
+        DATASET_ID = 's~FOO'
+        KIND = 'KIND'
+        INSIDE_NAME = 'IFOO'
+        OUTSIDE_NAME = 'OBAR'
+        INSIDE_VALUE = 1337
+
+        entity_inside = datastore_pb.Entity()
+        inside_prop = entity_inside.property.add()
+        inside_prop.name = INSIDE_NAME
+        inside_prop.value.integer_value = INSIDE_VALUE
+
+        entity_pb = datastore_pb.Entity()
+        entity_pb.key.partition_id.dataset_id = DATASET_ID
+        element = entity_pb.key.path_element.add()
+        element.kind = KIND
+
+        outside_prop = entity_pb.property.add()
+        outside_prop.name = OUTSIDE_NAME
+        outside_prop.value.entity_value.CopyFrom(entity_inside)
+
+        entity = self._callFUT(entity_pb)
+        self.assertEqual(entity.key.dataset_id, DATASET_ID)
+        self.assertEqual(entity.key.flat_path, (KIND,))
+        self.assertEqual(len(entity), 1)
+
+        inside_entity = entity[OUTSIDE_NAME]
+        self.assertEqual(inside_entity.key, None)
+        self.assertEqual(len(inside_entity), 1)
+        self.assertEqual(inside_entity[INSIDE_NAME], INSIDE_VALUE)
+
 
 class Test_key_from_protobuf(unittest2.TestCase):