Repository: googleapis/google-cloud-python

Issue:
Different meanings set on values when reading null
Hey,
we have a list in our datastore with ids and sometimes nulls in it. Like here:

```
["null","null","a953c730fc2e4647b3a5b4e88fa4f0ce","59ce754ddf03401eb09abd50c977cf67","null"]    
```

When we try to fetch them, we get the "Different meanings set on values within a list_value" Exception.

Is this a bug, that the method does not handles null?

[`_get_meaning` Method](https://github.com/GoogleCloudPlatform/gcloud-python/blob/efc1d41d230330fdd2f3c5c8df493b1ffb654be5/gcloud/datastore/helpers.py#L73):

``` python
def _get_meaning(value_pb, is_list=False):
    meaning = None
    if is_list:
        # An empty list will have no values, hence no shared meaning
        # set among them.
        if len(value_pb.list_value) == 0:
            return None

        # We check among all the meanings, some of which may be None,
        # the rest which may be enum/int values.
        all_meanings = set(_get_meaning(sub_value_pb)
                           for sub_value_pb in value_pb.list_value)
        meaning = all_meanings.pop()
        # The value we popped off should have been unique. If not
        # then we can't handle a list with values that have more
        # than one meaning.
        if all_meanings:
            raise ValueError('Different meanings set on values '
                             'within a list_value')
    elif value_pb.meaning:  # Simple field (int32)
        meaning = value_pb.meaning

    return meaning
```



Discussion:
Thanks for the report. This is at the moment a "feature not a bug", though we are certainly open to ideas / discussion.

@pcostell WDYT? Also an aside, how does one put a null in a list before `v1beta3`?

We managed to get null into lists by saving them from app-engine using python ndb.
On app-engine it is no problem to save or load these entities.

Also, for context, `_get_meaning` and associated code were added in #1241.

Also [original design discussion](https://github.com/GoogleCloudPlatform/gcloud-python/issues/1065#issuecomment-155577839) (go to comment, issue is long). I'm trying to track down the discussion of the decision to use a single type for an entire list, but can't find it.

@xamoom-raphael Can you provide a tiny `ndb` snippet? I'll store my own data with the remote API and then check the raw output to see what would need to be done to accommodate.

This is a screenshot from our entity in datastore.
![edit_entity_-_xamoom-tricia](https://cloud.githubusercontent.com/assets/10596108/13988042/878455f2-f10a-11e5-87eb-90a20348fb57.png)

And here is a shortened example of what we are doing on app-engine.
The model `LocalizedContentInformation` has a repeated property containing `ContentBlock` models. Some of these nested models have a `file_id` but some don't. This results in lists containing nulls for `file_id`. Which works fine on app-engine, but causes this error if we want to access this data using gcloud python client on compute-engine.

``` python
class ContentBlock(ndb.Model):
    content_block_type = ndb.IntegerProperty(required=True)​
    public = ndb.BooleanProperty(required=True)
    title = ndb.StringProperty()
    text = ndb.TextProperty()
    file_id = ndb.StringProperty()
​
class LocalizedContentInformation(ndb.Model):
    content_blocks = ndb.StructuredProperty(ContentBlock,repeated=True)
```

Thanks. I'll try to dig into it today (I am running a training so I may not get to it until the evening).

This was possible in the the private GAE proto for Datastore which stores properties as a repeated proto field. To write a list you just write a property with the same name multiple times and if you don't put any value it is considered null.

It seems like we should allow heterogeneous lists to be read since we can store them in Datastore.

OK I [reproduced](https://gist.github.com/dhermes/054fac0d3a4147ac96d5) the example.

@pcostell I / we made the decision not to allow heterogeneous meanings since it didn't seem possible within GAE. (We may also require everything in a list to be indexed / not-indexed for the same reason.) It's unclear to me if we should allow null as a special case or if we should just make it a free-for-all.

If we did try to do this, it'd be hard to get right:
- Retrieve a list with 4 entries: `[1, 2, 3, 4]` which has meanings `[15, None, None, 7]`
- The user doesn't even realize there are meanings
- The user changes the list to `[1, 2, 4, 7, 11]` but the `_meanings` on the `Entity` is unchanged (we consider them to be there just for compat, so not user facing)
- How do `[1, 2, 4, 7, 11]` and `[15, None, None, 7]` play nice when it's time to make a `put()` back to the server?

It seems like meanings and indexing information should follow the same
rules as if the user used protos directly. In this case they would be
linked to the value, so changing the individual value would clear it, or
adding a new value would have no meaning / default indexing info.

On Wed, Mar 23, 2016, 6:05 PM Danny Hermes notifications@github.com wrote:

> OK I reproduced https://gist.github.com/dhermes/054fac0d3a4147ac96d5
> the example.
> 
> @pcostell https://github.com/pcostell I / we made the decision not to
> allow heterogeneous meanings since it didn't seem possible within GAE. (We
> may also require everything in a list to be indexed / not-indexed for the
> same reason.) It's unclear to me if we should allow null as a special case
> or if we should just make it a free-for-all.
> 
> If we did try to do this, it'd be hard to get right:
> - Retrieve a list with 4 entries: [1, 2, 3, 4] which has meanings [15,
>   None, None, 7]
> - The user doesn't even realize there are meanings
> - The user changes the list to [1, 2, 4, 7, 11] but the _meanings on
>   the Entity is unchanged (we consider them to be there just for compat,
>   so not user facing)
> - How do [1, 2, 4, 7, 11] and [15, None, None, 7] play nice when it's
>   time to make a put() back to the server?
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly or view it on GitHub
> https://github.com/GoogleCloudPlatform/gcloud-python/issues/1649#issuecomment-200603311

Hey guys, can you give me an update on the issue.
Will there be a fix/change in the next version of gcloud-python?

@dhermes : Update?

No update right now. This needs a discussion though before we commit to any changes. At a glance, the suggestion from @pcostell of following the same rules as the protos may make it hard to present a higher-level interface to users.

@dhermes What can we do to make movement here? 

@xamoom-raphael : Just a heads up, the conversation here has gotten a bit theoretical when I think you just want to be able to read your data. Sorry about that.

Hang tight while we try to figure something out... If you have any ideas, chime in, but we have to be careful that whatever we do to deal with this situation doesn't give a "false sense of security" or overly complicate things under the hood.

---

I'm joining the bandwagon of "not sure how we could do this in a way that makes sense across the library".

Some thoughts...
- @pcostell : Is there a list of meanings somewhere? The whole magic number thing should probably be documented, right?
- AFAIK, our goal was to make a library that would work equally well across run-time environments (GCE, EC2, GAE, etc). I don't think this is the same thing as  "able to do everything NDB can do exactly as it does it".
- If we find a list of values, each with different meanings, we can always _not_ error out, but this would mean that re-saving that entity will lose the meanings, which to me seems like the worse version as it mucks with data unintentionally.

I can think of a couple different options:
1. **Keep track of meanings somewhere, and then put them back when we save.**
   This seems like a lot of extra complexity for something that's described in our docs as "The meaning field should only be populated for backwards compatibility." We already support this field for the common cases, but taking on nested entities with heterogeneous meanings under the hood seems... tricky and of debatable value.
2. **Error on save rather than load.**
   Instead of throwing an error when loading a protobuf, throw an error when you try to save an entity that was loaded from a proto containing values with heterogeneous meanings. (In other words, we keep a flag saying whether we run into this condition, and saving requires an explicit `clear_meanings=True` when saving the entity or errors out.)
3. **Add a flag to ignore meanings**
   Allow users to specify a flag saying that we should "ignore meanings" generally in Datastore, with a big huge warning saying "this clears out meanings from data you stored in App Engine" and links to a doc about the very specific use case you might have for flipping this flag.

The `meaning` field is documented as existing only to provide backwards compatibility (with App Engine). So we explicitly don't want any new libraries from explicitly populating it, although we do want to maintain existing set fields for compatibility.

@pcostell @jgeewax I think I have an idea how to do this. Starting now. (Sorry I meant to give this a shot last week.)


Resolving patch:
diff --git a/gcloud/datastore/helpers.py b/gcloud/datastore/helpers.py
--- a/gcloud/datastore/helpers.py
+++ b/gcloud/datastore/helpers.py
@@ -18,6 +18,7 @@
 """
 
 import datetime
+import itertools
 
 from google.protobuf import struct_pb2
 from google.type import latlng_pb2
@@ -45,10 +46,9 @@ def _get_meaning(value_pb, is_list=False):
 
     :rtype: int
     :returns: The meaning for the ``value_pb`` if one is set, else
-              :data:`None`.
-    :raises: :class:`ValueError <exceptions.ValueError>` if a list value
-             has disagreeing meanings (in sub-elements) or has some
-             elements with meanings and some without.
+              :data:`None`. For a list value, if there are disagreeing
+              means it just returns a list of meanings. If all the
+              list meanings agree, it just condenses them.
     """
     meaning = None
     if is_list:
@@ -59,15 +59,15 @@ def _get_meaning(value_pb, is_list=False):
 
         # We check among all the meanings, some of which may be None,
         # the rest which may be enum/int values.
-        all_meanings = set(_get_meaning(sub_value_pb)
-                           for sub_value_pb in value_pb.array_value.values)
-        meaning = all_meanings.pop()
-        # The value we popped off should have been unique. If not
-        # then we can't handle a list with values that have more
-        # than one meaning.
-        if all_meanings:
-            raise ValueError('Different meanings set on values '
-                             'within an array_value')
+        all_meanings = [_get_meaning(sub_value_pb)
+                        for sub_value_pb in value_pb.array_value.values]
+        unique_meanings = set(all_meanings)
+        if len(unique_meanings) == 1:
+            # If there is a unique meaning, we preserve it.
+            meaning = unique_meanings.pop()
+        else:  # We know len(value_pb.array_value.values) > 0.
+            # If the meaning is not unique, just return all of them.
+            meaning = all_meanings
     elif value_pb.meaning:  # Simple field (int32)
         meaning = value_pb.meaning
 
@@ -155,6 +155,48 @@ def entity_from_protobuf(pb):
     return entity
 
 
+def _set_pb_meaning_from_entity(entity, name, value, value_pb,
+                                is_list=False):
+    """Add meaning information (from an entity) to a protobuf.
+
+    :type entity: :class:`gcloud.datastore.entity.Entity`
+    :param entity: The entity to be turned into a protobuf.
+
+    :type name: string
+    :param name: The name of the property.
+
+    :type value: object
+    :param value: The current value stored as property ``name``.
+
+    :type value_pb: :class:`gcloud.datastore._generated.entity_pb2.Value`
+    :param value_pb: The protobuf value to add meaning / meanings to.
+
+    :type is_list: bool
+    :param is_list: (Optional) Boolean indicating if the ``value`` is
+                    a list value.
+    """
+    if name not in entity._meanings:
+        return
+
+    meaning, orig_value = entity._meanings[name]
+    # Only add the meaning back to the protobuf if the value is
+    # unchanged from when it was originally read from the API.
+    if orig_value is not value:
+        return
+
+    # For lists, we set meaning on each sub-element.
+    if is_list:
+        if not isinstance(meaning, list):
+            meaning = itertools.repeat(meaning)
+        val_iter = six.moves.zip(value_pb.array_value.values,
+                                 meaning)
+        for sub_value_pb, sub_meaning in val_iter:
+            if sub_meaning is not None:
+                sub_value_pb.meaning = sub_meaning
+    else:
+        value_pb.meaning = meaning
+
+
 def entity_to_protobuf(entity):
     """Converts an entity into a protobuf.
 
@@ -187,17 +229,8 @@ def entity_to_protobuf(entity):
                 sub_value.exclude_from_indexes = True
 
         # Add meaning information to protobuf.
-        if name in entity._meanings:
-            meaning, orig_value = entity._meanings[name]
-            # Only add the meaning back to the protobuf if the value is
-            # unchanged from when it was originally read from the API.
-            if orig_value is value:
-                # For lists, we set meaning on each sub-element.
-                if value_is_list:
-                    for sub_value_pb in value_pb.array_value.values:
-                        sub_value_pb.meaning = meaning
-                else:
-                    value_pb.meaning = meaning
+        _set_pb_meaning_from_entity(entity, name, value, value_pb,
+                                    is_list=value_is_list)
 
     return entity_pb
 


Test patch:
diff --git a/gcloud/datastore/test_helpers.py b/gcloud/datastore/test_helpers.py
--- a/gcloud/datastore/test_helpers.py
+++ b/gcloud/datastore/test_helpers.py
@@ -341,6 +341,32 @@ def test_meaning_with_change(self):
         #       value stored.
         self._compareEntityProto(entity_pb, expected_pb)
 
+    def test_variable_meanings(self):
+        from gcloud.datastore._generated import entity_pb2
+        from gcloud.datastore.entity import Entity
+        from gcloud.datastore.helpers import _new_value_pb
+
+        entity = Entity()
+        name = 'quux'
+        entity[name] = values = [1, 20, 300]
+        meaning = 9
+        entity._meanings[name] = ([None, meaning, None], values)
+        entity_pb = self._callFUT(entity)
+
+        # Construct the expected protobuf.
+        expected_pb = entity_pb2.Entity()
+        value_pb = _new_value_pb(expected_pb, name)
+        value0 = value_pb.array_value.values.add()
+        value0.integer_value = values[0]
+        # The only array entry with a meaning is the middle one.
+        value1 = value_pb.array_value.values.add()
+        value1.integer_value = values[1]
+        value1.meaning = meaning
+        value2 = value_pb.array_value.values.add()
+        value2.integer_value = values[2]
+
+        self._compareEntityProto(entity_pb, expected_pb)
+
 
 class Test_key_from_protobuf(unittest2.TestCase):
 
@@ -813,7 +839,7 @@ def test_array_value(self):
         result = self._callFUT(value_pb, is_list=True)
         self.assertEqual(meaning, result)
 
-    def test_array_value_disagreeing(self):
+    def test_array_value_multiple_meanings(self):
         from gcloud.datastore._generated import entity_pb2
 
         value_pb = entity_pb2.Value()
@@ -827,10 +853,10 @@ def test_array_value_disagreeing(self):
         sub_value_pb1.string_value = u'hi'
         sub_value_pb2.string_value = u'bye'
 
-        with self.assertRaises(ValueError):
-            self._callFUT(value_pb, is_list=True)
+        result = self._callFUT(value_pb, is_list=True)
+        self.assertEqual(result, [meaning1, meaning2])
 
-    def test_array_value_partially_unset(self):
+    def test_array_value_meaning_partially_unset(self):
         from gcloud.datastore._generated import entity_pb2
 
         value_pb = entity_pb2.Value()
@@ -842,8 +868,8 @@ def test_array_value_partially_unset(self):
         sub_value_pb1.string_value = u'hi'
         sub_value_pb2.string_value = u'bye'
 
-        with self.assertRaises(ValueError):
-            self._callFUT(value_pb, is_list=True)
+        result = self._callFUT(value_pb, is_list=True)
+        self.assertEqual(result, [meaning1, None])
 
 
 class TestGeoPoint(unittest2.TestCase):