Repository: googleapis/google-cloud-python

Issue:
GAX can't log nested structs
This is the root issue in #2521 , if we feel it's the same issue can just rename that one. But when using gax, 

``` python
from google.cloud import logging
client = logging.Client()
logger = client.logger('errors')

payload = {
    'a': {
        'b': 'c',
    }
}
logger.log_struct(payload)
```

produces: 

``` python
  File "e.py", line 22, in <module>
    logger.log_struct(payload)
  File "/Users/waprin/.virtualenvs/tasks/lib/python2.7/site-packages/google/cloud/logging/logger.py", line 226, in log_struct
    client.logging_api.write_entries([entry_resource])
  File "/Users/waprin/.virtualenvs/tasks/lib/python2.7/site-packages/google/cloud/logging/_gax.py", line 108, in write_entries
    entry_pbs = [_log_entry_mapping_to_pb(entry) for entry in entries]
  File "/Users/waprin/.virtualenvs/tasks/lib/python2.7/site-packages/google/cloud/logging/_gax.py", line 607, in _log_entry_mapping_to_pb
    entry_pb.json_payload[key] = value
  File "/Users/waprin/.virtualenvs/tasks/lib/python2.7/site-packages/google/protobuf/internal/well_known_types.py", line 732, in __setitem__
    _SetStructValue(self.fields[key], value)
  File "/Users/waprin/.virtualenvs/tasks/lib/python2.7/site-packages/google/protobuf/internal/well_known_types.py", line 702, in _SetStructValue
    raise ValueError('Unexpected type')
ValueError: Unexpected type
```



Discussion:
@dhermes I don't think this is actually a GAX/gRPC issue:  our own `_log_entry_mapping_to_pb` function is likely at fault.

It looks like [this](https://github.com/GoogleCloudPlatform/google-cloud-python/blob/0aca3f6ed05f3340333f231445a3d8b17d8d8e6c/logging/google/cloud/logging/logger.py#L136) causes `jsonPayload` to be set for when `_log_entry_mapping_to_pb()` is called later.

[This](https://github.com/GoogleCloudPlatform/google-cloud-python/blob/0aca3f6ed05f3340333f231445a3d8b17d8d8e6c/logging/google/cloud/logging/_gax.py#L605) mapping in `_log_entry_mapping_to_pb()` should probably either be recursive or support some number of levels.

``` python
    if 'jsonPayload' in mapping:
        for key, value in mapping['jsonPayload'].items():
            entry_pb.json_payload[key] = value
```

@dhermes & @waprin WDYT of this over #2553?

SGTM. Bit surprising that that recursively setting a struct needs to be done manually, still trying to understand it.

@waprin, it looks like the [generated code side](https://github.com/google/protobuf/blob/008b5a228b37c054f46ba478ccafa5e855cb16db/python/google/protobuf/internal/well_known_types.py#L690-L702) only accepts bool, text, and number types.

However the docs make it sound like `jsonPayload` should just accept the object.
https://cloud.google.com/logging/docs/api/ref_v2beta1/rest/v2beta1/LogEntry

Maybe the payload needs to be serialized first?

@bjwatson do you know if this is a bug we should add on googleapis/googleapis or if `jsonPayload` needs to be converted to a string first?

@daspecster I think this is because `json_payload` is not defined in the [proto](https://github.com/googleapis/googleapis/blob/master/google/logging/v2/log_entry.proto#L67) as `map<string, string>` like `labels` is. Instead it is defined as [`google.protobuf.Struct`](https://github.com/google/protobuf/blob/master/src/google/protobuf/struct.proto#L52), which adds one more level to the object hierarchy.

Change [this code](https://github.com/GoogleCloudPlatform/google-cloud-python/blob/0aca3f6ed05f3340333f231445a3d8b17d8d8e6c/logging/google/cloud/logging/_gax.py#L605):

``` python
    if 'jsonPayload' in mapping:
        for key, value in mapping['jsonPayload'].items():
            entry_pb.json_payload[key] = value
```

To something like:

``` python
    if 'jsonPayload' in mapping:
        fields = {}
        for key, value in mapping['jsonPayload'].items():
            fields[key] = value
        entry_pb.json_payload = google.protobuf.Struct(fields)
```

I flagged this as an opportunity to improve the `LoggingServiceV2Api.write_log_entries()` method sample in the GAPIC code: https://github.com/googleapis/toolkit/issues/636

Thanks @bjwatson!

@bjwatson What's the type of `mapping['jsonPayload']`? Seems like

``` python
        fields = {}
        for key, value in mapping['jsonPayload'].items():
            fields[key] = value
```

is a really long way of saying:

``` python
fields = mapping['jsonPayload'].copy()
```

@dhermes I'm guessing it's a dictionary. It's defined in the gcloud code: https://github.com/GoogleCloudPlatform/google-cloud-python/blob/0aca3f6ed05f3340333f231445a3d8b17d8d8e6c/logging/google/cloud/logging/logger.py#L223

Thanks for the tip on the `.copy()` function. Seems like a good simplification if it works.

@bjwatson @dhermes still having a bit of trouble getting this right. If I try the suggested solution I get:

```
 entry_pb.json_payload = Struct(fields)
TypeError: init() takes exactly 1 argument (2 given)
```

Doesn't seem like I can just construct Struct with a dict. So then I tried:

```
entry_pb.json_payload = Struct()
for key, value in mapping['jsonValue']:
    entry_pb.json_payload[key] = value
```

I get 

```
ValueError: May not set values directly, call my_map[key].foo = 5
```

I see this in [the docs](https://developers.google.com/protocol-buffers/docs/reference/python-generated)

> As with embedded message fields, messages cannot be directly assigned into a map value. Instead, to add a message as a map value you reference an undefined key, which constructs and returns a new submessage:

So I see in the [Struct proto](https://github.com/google/protobuf/blob/master/src/google/protobuf/struct.proto) it is a `map<string, Value>`. So since Value is a Message I need to actually set its fields explicitly like:

```
entry_pb.json_payload[key].string_value = "whatever"
```

Unfortunately the `Value` is not always a string and so `string_value` is not always appropriate. We could cast the value to a string but that doesn't seem right as it would be changing the types. What I think I really want is a function that recursively goes through the dict and uses the Python types it sees to set the correct union type for the Value (string_value, struct_value, etc). But it really feels like that code should be in some sort of library and not in this logging specific file. We could throw it in core helpers at a minimum, but maybe this function exists somewhere else already? 

Wondering if I'm missing something?

w00t:

``` python
>>> import json
>>> from google.protobuf.json_format import Parse
>>> from google.protobuf.struct_pb2 import Struct
>>> msg = Struct()
>>> result = Parse(json.dumps({'a': {'b': 'c'}}), msg)
>>> result is msg
True
>>> msg
fields {
  key: "a"
  value {
    struct_value {
      fields {
        key: "b"
        value {
          string_value: "c"
        }
      }
    }
  }
}
```


Resolving patch:
diff --git a/logging/google/cloud/logging/_gax.py b/logging/google/cloud/logging/_gax.py
--- a/logging/google/cloud/logging/_gax.py
+++ b/logging/google/cloud/logging/_gax.py
@@ -14,8 +14,6 @@
 
 """GAX wrapper for Logging API requests."""
 
-import json
-
 from google.gax import CallOptions
 from google.gax import INITIAL_PAGE
 from google.gax.errors import GaxError
@@ -24,7 +22,7 @@
 from google.logging.v2.logging_config_pb2 import LogSink
 from google.logging.v2.logging_metrics_pb2 import LogMetric
 from google.logging.v2.log_entry_pb2 import LogEntry
-from google.protobuf.json_format import Parse
+from google.protobuf.json_format import ParseDict
 from grpc import StatusCode
 
 from google.cloud._helpers import _datetime_to_pb_timestamp
@@ -603,11 +601,10 @@ def _log_entry_mapping_to_pb(mapping):
             entry_pb.labels[key] = value
 
     if 'jsonPayload' in mapping:
-        for key, value in mapping['jsonPayload'].items():
-            entry_pb.json_payload[key] = value
+        ParseDict(mapping['jsonPayload'], entry_pb.json_payload)
 
     if 'protoPayload' in mapping:
-        Parse(json.dumps(mapping['protoPayload']), entry_pb.proto_payload)
+        ParseDict(mapping['protoPayload'], entry_pb.proto_payload)
 
     if 'httpRequest' in mapping:
         _http_request_mapping_to_pb(


Test patch:
diff --git a/logging/unit_tests/test__gax.py b/logging/unit_tests/test__gax.py
--- a/logging/unit_tests/test__gax.py
+++ b/logging/unit_tests/test__gax.py
@@ -80,16 +80,12 @@ def test_list_entries_no_paging(self):
         self.assertEqual(page_size, 0)
         self.assertIs(options.page_token, INITIAL_PAGE)
 
-    def test_list_entries_with_paging(self):
-        from google.protobuf.struct_pb2 import Value
+    def _list_entries_with_paging_helper(self, payload, struct_pb):
         from google.cloud._testing import _GAXPageIterator
+
         SIZE = 23
         TOKEN = 'TOKEN'
         NEW_TOKEN = 'NEW_TOKEN'
-        PAYLOAD = {'message': 'MESSAGE', 'weather': 'sunny'}
-        struct_pb = _StructPB({
-            key: Value(string_value=value) for key, value in PAYLOAD.items()
-        })
         response = _GAXPageIterator(
             [_LogEntryPB(self.LOG_NAME, json_payload=struct_pb)], NEW_TOKEN)
         gax_api = _GAXLoggingAPI(_list_log_entries_response=response)
@@ -103,7 +99,7 @@ def test_list_entries_with_paging(self):
         self.assertIsInstance(entry, dict)
         self.assertEqual(entry['logName'], self.LOG_NAME)
         self.assertEqual(entry['resource'], {'type': 'global'})
-        self.assertEqual(entry['jsonPayload'], PAYLOAD)
+        self.assertEqual(entry['jsonPayload'], payload)
         self.assertEqual(next_token, NEW_TOKEN)
 
         projects, filter_, order_by, page_size, options = (
@@ -114,6 +110,43 @@ def test_list_entries_with_paging(self):
         self.assertEqual(page_size, SIZE)
         self.assertEqual(options.page_token, TOKEN)
 
+    def test_list_entries_with_paging(self):
+        from google.protobuf.struct_pb2 import Struct
+        from google.protobuf.struct_pb2 import Value
+
+        payload = {'message': 'MESSAGE', 'weather': 'sunny'}
+        struct_pb = Struct(fields={
+            key: Value(string_value=value) for key, value in payload.items()
+        })
+        self._list_entries_with_paging_helper(payload, struct_pb)
+
+    def test_list_entries_with_paging_nested_payload(self):
+        from google.protobuf.struct_pb2 import Struct
+        from google.protobuf.struct_pb2 import Value
+
+        payload = {}
+        struct_fields = {}
+        # Add a simple key.
+        key = 'message'
+        payload[key] = 'MESSAGE'
+        struct_fields[key] = Value(string_value=payload[key])
+        # Add a nested key.
+        key = 'weather'
+        sub_value = {}
+        sub_fields = {}
+        sub_key = 'temperature'
+        sub_value[sub_key] = 75
+        sub_fields[sub_key] = Value(number_value=sub_value[sub_key])
+        sub_key = 'precipitation'
+        sub_value[sub_key] = False
+        sub_fields[sub_key] = Value(bool_value=sub_value[sub_key])
+        # Update the parent payload.
+        payload[key] = sub_value
+        struct_fields[key] = Value(struct_value=Struct(fields=sub_fields))
+        # Make the struct_pb for our dict.
+        struct_pb = Struct(fields=struct_fields)
+        self._list_entries_with_paging_helper(payload, struct_pb)
+
     def test_list_entries_with_extra_properties(self):
         from datetime import datetime
         from google.logging.type.log_severity_pb2 import WARNING
@@ -317,18 +350,16 @@ def test_write_entries_w_extra_properties(self):
         self.assertIsNone(options)
         # pylint: enable=too-many-statements
 
-    def test_write_entries_multiple(self):
+    def _write_entries_multiple_helper(self, json_payload, json_struct_pb):
         # pylint: disable=too-many-statements
         import datetime
         from google.logging.type.log_severity_pb2 import WARNING
         from google.logging.v2.log_entry_pb2 import LogEntry
         from google.protobuf.any_pb2 import Any
-        from google.protobuf.struct_pb2 import Struct
         from google.cloud._helpers import _datetime_to_rfc3339, UTC
         TEXT = 'TEXT'
         NOW = datetime.datetime.utcnow().replace(tzinfo=UTC)
         TIMESTAMP_TYPE_URL = 'type.googleapis.com/google.protobuf.Timestamp'
-        JSON = {'payload': 'PAYLOAD', 'type': 'json'}
         PROTO = {
             '@type': TIMESTAMP_TYPE_URL,
             'value': _datetime_to_rfc3339(NOW),
@@ -339,7 +370,7 @@ def test_write_entries_multiple(self):
         ENTRIES = [
             {'textPayload': TEXT,
              'severity': WARNING},
-            {'jsonPayload': JSON,
+            {'jsonPayload': json_payload,
              'operation': {'producer': PRODUCER, 'id': OPID}},
             {'protoPayload': PROTO,
              'httpRequest': {'requestUrl': URL}},
@@ -373,10 +404,7 @@ def test_write_entries_multiple(self):
         self.assertEqual(entry.log_name, '')
         self.assertEqual(entry.resource.type, '')
         self.assertEqual(entry.labels, {})
-        json_struct = entry.json_payload
-        self.assertIsInstance(json_struct, Struct)
-        self.assertEqual(json_struct.fields['payload'].string_value,
-                         JSON['payload'])
+        self.assertEqual(entry.json_payload, json_struct_pb)
         operation = entry.operation
         self.assertEqual(operation.producer, PRODUCER)
         self.assertEqual(operation.id, OPID)
@@ -399,6 +427,44 @@ def test_write_entries_multiple(self):
         self.assertIsNone(options)
         # pylint: enable=too-many-statements
 
+    def test_write_entries_multiple(self):
+        from google.protobuf.struct_pb2 import Struct
+        from google.protobuf.struct_pb2 import Value
+
+        json_payload = {'payload': 'PAYLOAD', 'type': 'json'}
+        json_struct_pb = Struct(fields={
+            key: Value(string_value=value)
+            for key, value in json_payload.items()
+        })
+        self._write_entries_multiple_helper(json_payload, json_struct_pb)
+
+    def test_write_entries_multiple_nested_payload(self):
+        from google.protobuf.struct_pb2 import Struct
+        from google.protobuf.struct_pb2 import Value
+
+        json_payload = {}
+        struct_fields = {}
+        # Add a simple key.
+        key = 'hello'
+        json_payload[key] = 'me you looking for'
+        struct_fields[key] = Value(string_value=json_payload[key])
+        # Add a nested key.
+        key = 'everything'
+        sub_value = {}
+        sub_fields = {}
+        sub_key = 'answer'
+        sub_value[sub_key] = 42
+        sub_fields[sub_key] = Value(number_value=sub_value[sub_key])
+        sub_key = 'really?'
+        sub_value[sub_key] = False
+        sub_fields[sub_key] = Value(bool_value=sub_value[sub_key])
+        # Update the parent payload.
+        json_payload[key] = sub_value
+        struct_fields[key] = Value(struct_value=Struct(fields=sub_fields))
+        # Make the struct_pb for our dict.
+        json_struct_pb = Struct(fields=struct_fields)
+        self._write_entries_multiple_helper(json_payload, json_struct_pb)
+
     def test_logger_delete(self):
         LOG_PATH = 'projects/%s/logs/%s' % (self.PROJECT, self.LOG_NAME)
         gax_api = _GAXLoggingAPI()
@@ -1057,12 +1123,6 @@ def __init__(self, type_='global', **labels):
         self.labels = labels
 
 
-class _StructPB(object):
-
-    def __init__(self, fields):
-        self.fields = fields
-
-
 class _LogEntryPB(object):
 
     severity = 0
diff --git a/system_tests/logging_.py b/system_tests/logging_.py
--- a/system_tests/logging_.py
+++ b/system_tests/logging_.py
@@ -57,6 +57,15 @@ def setUpModule():
 
 class TestLogging(unittest.TestCase):
 
+    JSON_PAYLOAD = {
+        'message': 'System test: test_log_struct',
+        'weather': {
+            'clouds': 'party or partly',
+            'temperature': 70,
+            'precipitation': False,
+        },
+    }
+
     def setUp(self):
         self.to_delete = []
         self._handlers_cache = logging.getLogger().handlers[:]
@@ -120,18 +129,14 @@ def test_log_text_w_metadata(self):
         self.assertEqual(request['status'], STATUS)
 
     def test_log_struct(self):
-        JSON_PAYLOAD = {
-            'message': 'System test: test_log_struct',
-            'weather': 'partly cloudy',
-        }
         logger = Config.CLIENT.logger(self._logger_name())
         self.to_delete.append(logger)
 
-        logger.log_struct(JSON_PAYLOAD)
+        logger.log_struct(self.JSON_PAYLOAD)
         entries, _ = self._list_entries(logger)
 
         self.assertEqual(len(entries), 1)
-        self.assertEqual(entries[0].payload, JSON_PAYLOAD)
+        self.assertEqual(entries[0].payload, self.JSON_PAYLOAD)
 
     def test_log_handler_async(self):
         LOG_MESSAGE = 'It was the worst of times'
@@ -145,12 +150,12 @@ def test_log_handler_async(self):
         cloud_logger.addHandler(handler)
         cloud_logger.warn(LOG_MESSAGE)
         entries, _ = self._list_entries(logger)
-        JSON_PAYLOAD = {
+        expected_payload = {
             'message': LOG_MESSAGE,
             'python_logger': handler.name
         }
         self.assertEqual(len(entries), 1)
-        self.assertEqual(entries[0].payload, JSON_PAYLOAD)
+        self.assertEqual(entries[0].payload, expected_payload)
 
     def test_log_handler_sync(self):
         LOG_MESSAGE = 'It was the best of times.'
@@ -169,12 +174,12 @@ def test_log_handler_sync(self):
         cloud_logger.warn(LOG_MESSAGE)
 
         entries, _ = self._list_entries(logger)
-        JSON_PAYLOAD = {
+        expected_payload = {
             'message': LOG_MESSAGE,
             'python_logger': LOGGER_NAME
         }
         self.assertEqual(len(entries), 1)
-        self.assertEqual(entries[0].payload, JSON_PAYLOAD)
+        self.assertEqual(entries[0].payload, expected_payload)
 
     def test_log_root_handler(self):
         LOG_MESSAGE = 'It was the best of times.'
@@ -188,19 +193,15 @@ def test_log_root_handler(self):
         logging.warn(LOG_MESSAGE)
 
         entries, _ = self._list_entries(logger)
-        JSON_PAYLOAD = {
+        expected_payload = {
             'message': LOG_MESSAGE,
             'python_logger': 'root'
         }
 
         self.assertEqual(len(entries), 1)
-        self.assertEqual(entries[0].payload, JSON_PAYLOAD)
+        self.assertEqual(entries[0].payload, expected_payload)
 
     def test_log_struct_w_metadata(self):
-        JSON_PAYLOAD = {
-            'message': 'System test: test_log_struct',
-            'weather': 'partly cloudy',
-        }
         INSERT_ID = 'INSERTID'
         SEVERITY = 'INFO'
         METHOD = 'POST'
@@ -214,12 +215,12 @@ def test_log_struct_w_metadata(self):
         logger = Config.CLIENT.logger(self._logger_name())
         self.to_delete.append(logger)
 
-        logger.log_struct(JSON_PAYLOAD, insert_id=INSERT_ID, severity=SEVERITY,
-                          http_request=REQUEST)
+        logger.log_struct(self.JSON_PAYLOAD, insert_id=INSERT_ID,
+                          severity=SEVERITY, http_request=REQUEST)
         entries, _ = self._list_entries(logger)
 
         self.assertEqual(len(entries), 1)
-        self.assertEqual(entries[0].payload, JSON_PAYLOAD)
+        self.assertEqual(entries[0].payload, self.JSON_PAYLOAD)
         self.assertEqual(entries[0].insert_id, INSERT_ID)
         self.assertEqual(entries[0].severity, SEVERITY)
         request = entries[0].http_request