Repository: googleapis/google-cloud-python

Issue:
Firestore: Out of expect when document update SERVER_TIMESTAMP and an arbitrary value by field path
#### Environment details

1. OS type and version:
Both happened on Windows 7 and Ubuntu 18.04
2. Python version:
python 3.6
3. google-cloud-<service> version:
firebase-admin                     2.14.0
google-api-core                    1.7.0
google-auth                        1.6.2
google-cloud-core                  0.29.1
google-cloud-firestore             0.31.0
google-cloud-storage               1.13.2
google-resumable-media             0.3.2
googleapis-common-protos           1.5.5

#### Reference
The following case is modified by the [firestore document update section](https://googleapis.github.io/google-cloud-python/latest/firestore/document.html#google.cloud.firestore_v1beta1.document.DocumentReference.update) example.

#### Steps to reproduce

1. To illustrate this, consider a document with
```
>>> snapshot = document.get()
>>> snapshot.to_dict()
{
    'foo': {
        'bar': 'baz',
    },
    'other': True,
}
```
stored on the server.

2. To set a field to the current time and another field to an arbitrary value(ex: "test"). We expect sending
```
>>> field_updates = {
...     'foo.now': firestore.SERVER_TIMESTAMP,
...     'foo.test': "test",
... }
>>> document.update(field_updates)
```
would update the value on the server to:
```
>>> snapshot = document.get()
>>> snapshot.to_dict()
{
    'foo': {
        'bar': 'baz',
        'now': datetime.datetime(2019, ...),
        'test': 'test'
    },
    'other': True,
}
```
3. However, in the real case, the value on the server are:
 ```
>>> snapshot = document.get()
>>> snapshot.to_dict()
{
    'foo': {
        'now': datetime.datetime(2019, ...),
        'test': 'test'
    },
    'other': True,
}
```
4. Apparently, the previous field 'foo.bar' disappear, but it doesn't match our expect.

#### Code example

##### Step 1: prepare the data
```python
import firebase_admin
from firebase_admin import credentials, firestore

cred = credentials.Certificate('your-api-key.json')
firebase_admin.initialize_app(cred)

db = firestore.client()
document = db.collection('test').document()
document.set({
    "foo":{
        "bar": 'baz'
    },
    'other': True
})
snapshot = document.get()
print(snapshot.to_dict())
```
console:
```
{
    'foo': {
        'bar': 'baz',
    },
    'other': True,
}
```

##### Step 2: update the data
```python
field_updates = {
    'foo.test': 'test',
    'foo.now': firestore.SERVER_TIMESTAMP,
}
document.update(field_updates)

snapshot = document.get()
print(snapshot.to_dict())
```
console:
```
{
    'foo': {
        'now': datetime.datetime(2019, ...),
        'test': 'test'
    },
    'other': True,
}
```

##### Expected output
```
{
    'foo': {
        'bar': 'baz',
        'now': datetime.datetime(2019, ...),
        'test': 'test'
    },
    'other': True,
}
```
Thank you.


Discussion:
Working to reproduce with [this gist](https://gist.github.com/tseaver/41144415460ad26c012359d657a32680):

```bash
$ python3.6 -m venv /tmp/gcp/7215
$ /tmp/gcp/7215/bin/pip install --upgrade setuptools pip
...
Successfully installed pip-19.0.1 setuptools-40.7.1
$ /tmp/gcp/7215/bin/pip install google-cloud-firestore
...
Successfully installed cachetools-3.0.0 certifi-2018.11.29 chardet-3.0.4 google-api-core-1.7.0 google-auth-1.6.2 google-cloud-core-0.29.1 google-cloud-firestore-0.31.0 googleapis-common-protos-1.5.6 grpcio-1.18.0 idna-2.8 protobuf-3.6.1 pyasn1-0.4.5 pyasn1-modules-0.2.4 pytz-2018.9 requests-2.21.0 rsa-4.0 six-1.12.0 urllib3-1.24.1
$ /tmp/gcp/7215/bin/python /tmp/gcp/7215/reproduce_7215.py

Prepared document: o8IQPjUXF4YgLMdjoaae
{'foo': {'bar': 'baz'}, 'other': True}

Updated document: o8IQPjUXF4YgLMdjoaae
{'foo': {'now': DatetimeWithNanoseconds(2019, 1, 29, 17, 18, 10, 477000, tzinfo=<UTC>),
         'test': 'test'},
 'other': True}

Deleted document: o8IQPjUXF4YgLMdjoaae


```
@fishballLin Thanks for the detailed report!
@tseaver If you need any help or further information, please contact with me any time.
Thank you.
OK, the `Write` protobufs being sent with the `CommitRequest` look like:

```python
[update {
  name: "projects/tseaver-firestore/databases/(default)/documents/test/hCpyH9RfMqgbSK1TjuNe"
  fields {
    key: "foo"
    value {
      map_value {
        fields {
          key: "test"
          value {
            string_value: "test"
          }
        }
      }
    }
  }
}
update_mask {
  field_paths: "foo"
  field_paths: "foo.test"
}
current_document {
  exists: true
}
,
 transform {
  document: "projects/tseaver-firestore/databases/(default)/documents/test/hCpyH9RfMqgbSK1TjuNe"
  field_transforms {
    field_path: "foo.now"
    set_to_server_value: REQUEST_TIME
  }
}
```

Which is why `foo['bar']` is getting deleted:  having `foo` in the `update_mask.field_paths` means that `foo` is supposed to be replaced with the contents of the `foo` key in the `update` message.

We are generating it that way on account of a requirement that we include the "parent" key in the update mask of any nested transform.

@jadekler I think this may be a hole in our conformance testing:  can you verify that the `foo.bar` field should *not* be erased when setting `foo.now` to the server timestamp in an update request?
@jadekler I've created a [`textproto` testcase](https://gist.github.com/tseaver/edf79a246de4c46dd3d99d2d35596234) which passes on the current Python implementation, but exposes this (unexpected) behavior.
@tseaver https://github.com/googleapis/google-cloud-common/pull/275

Resolving patch:
diff --git a/firestore/google/cloud/firestore_v1/_helpers.py b/firestore/google/cloud/firestore_v1/_helpers.py
--- a/firestore/google/cloud/firestore_v1/_helpers.py
+++ b/firestore/google/cloud/firestore_v1/_helpers.py
@@ -836,10 +836,6 @@ def _get_update_mask(self, allow_empty_mask=False):
         for field_path in self.top_level_paths:
             if field_path not in self.transform_paths:
                 mask_paths.append(field_path.to_api_repr())
-            else:
-                prefix = FieldPath(*field_path.parts[:-1])
-                if prefix.parts:
-                    mask_paths.append(prefix.to_api_repr())
 
         return common_pb2.DocumentMask(field_paths=mask_paths)
 
diff --git a/firestore/google/cloud/firestore_v1beta1/_helpers.py b/firestore/google/cloud/firestore_v1beta1/_helpers.py
--- a/firestore/google/cloud/firestore_v1beta1/_helpers.py
+++ b/firestore/google/cloud/firestore_v1beta1/_helpers.py
@@ -785,10 +785,6 @@ def _get_update_mask(self, allow_empty_mask=False):
         for field_path in self.top_level_paths:
             if field_path not in self.transform_paths:
                 mask_paths.append(field_path.to_api_repr())
-            else:
-                prefix = FieldPath(*field_path.parts[:-1])
-                if prefix.parts:
-                    mask_paths.append(prefix.to_api_repr())
 
         return common_pb2.DocumentMask(field_paths=mask_paths)
 


Test patch:
diff --git a/firestore/tests/unit/v1/test__helpers.py b/firestore/tests/unit/v1/test__helpers.py
--- a/firestore/tests/unit/v1/test__helpers.py
+++ b/firestore/tests/unit/v1/test__helpers.py
@@ -2110,10 +2110,7 @@ def _helper(self, option=None, do_transform=False, **write_kwargs):
 
         map_pb = document_pb2.MapValue(fields={"yum": _value_pb(bytes_value=value)})
 
-        if do_transform:
-            field_paths = [field_path1, "blog"]
-        else:
-            field_paths = [field_path1]
+        field_paths = [field_path1]
 
         expected_update_pb = write_pb2.Write(
             update=document_pb2.Document(
diff --git a/firestore/tests/unit/v1/testdata/update-nested-transform-and-nested-value.json b/firestore/tests/unit/v1/testdata/update-nested-transform-and-nested-value.json
new file mode 100644
--- /dev/null
+++ b/firestore/tests/unit/v1/testdata/update-nested-transform-and-nested-value.json
@@ -0,0 +1,52 @@
+{
+  "tests": [
+    {
+      "description": "update: Nested transforms should not affect the field mask, even\nwhen there are other values that do. Transforms should only affect the\nDocumentTransform_FieldTransform list.",
+      "comment": "For updates, top-level paths in json-like map inputs\nare split on the dot. That is, an input {\"a.b.c\": 7} results in an update to\nfield c of object b of object a with value 7. In order to specify this behavior,\nthe update must use a fieldmask \"a.b.c\". However, fieldmasks are only used for\nconcrete values - transforms are separately encoded in a\nDocumentTransform_FieldTransform array.\n\nThis test exercises a bug found in python (https://github.com/googleapis/google-cloud-python/issues/7215)\nin which nested transforms ({\"a.c\": \"ServerTimestamp\"}) next to nested values\n({\"a.b\": 7}) incorrectly caused the fieldmask \"a\" to be set, which has the\neffect of wiping out all data in \"a\" other than what was specified in the\njson-like input.\n\nInstead, as this test specifies, transforms should not affect the fieldmask.",
+      "update": {
+        "docRefPath": "projects/projectID/databases/(default)/documents/C/d",
+        "jsonData": "{\"a.b\": 7, \"a.c\": \"ServerTimestamp\"}",
+        "request": {
+          "database": "projects/projectID/databases/(default)",
+          "writes": [
+            {
+              "update": {
+                "name": "projects/projectID/databases/(default)/documents/C/d",
+                "fields": {
+                  "a": {
+                    "mapValue": {
+                      "fields": {
+                        "b": {
+                          "integerValue": "7"
+                        }
+                      }
+                    }
+                  }
+                }
+              },
+              "updateMask": {
+                "fieldPaths": [
+                  "a.b"
+                ]
+              },
+              "currentDocument": {
+                "exists": true
+              }
+            },
+            {
+              "transform": {
+                "document": "projects/projectID/databases/(default)/documents/C/d",
+                "fieldTransforms": [
+                  {
+                    "fieldPath": "a.c",
+                    "setToServerValue": "REQUEST_TIME"
+                  }
+                ]
+              }
+            }
+          ]
+        }
+      }
+    }
+  ]
+}
diff --git a/firestore/tests/unit/v1beta1/test__helpers.py b/firestore/tests/unit/v1beta1/test__helpers.py
--- a/firestore/tests/unit/v1beta1/test__helpers.py
+++ b/firestore/tests/unit/v1beta1/test__helpers.py
@@ -1801,10 +1801,7 @@ def _helper(self, option=None, do_transform=False, **write_kwargs):
 
         map_pb = document_pb2.MapValue(fields={"yum": _value_pb(bytes_value=value)})
 
-        if do_transform:
-            field_paths = [field_path1, "blog"]
-        else:
-            field_paths = [field_path1]
+        field_paths = [field_path1]
 
         expected_update_pb = write_pb2.Write(
             update=document_pb2.Document(
diff --git a/firestore/tests/unit/v1beta1/testdata/update-nested-transform-and-nested-value.textproto b/firestore/tests/unit/v1beta1/testdata/update-nested-transform-and-nested-value.textproto
new file mode 100644
--- /dev/null
+++ b/firestore/tests/unit/v1beta1/testdata/update-nested-transform-and-nested-value.textproto
@@ -0,0 +1,58 @@
+# DO NOT MODIFY. This file was generated by
+# github.com/GoogleCloudPlatform/google-cloud-common/testing/firestore/cmd/generate-firestore-tests/generate-firestore-tests.go.
+
+# For updates, top-level paths in json-like map inputs are split on the dot. That
+# is, an input {"a.b.c": 7} results in an update to field c of object b of object
+# a with value 7. In order to specify this behavior, the update must use a
+# fieldmask "a.b.c". However, fieldmasks are only used for concrete values -
+# transforms are separately encoded in a DocumentTransform_FieldTransform array.
+
+# This test exercises a bug found in python
+# (https://github.com/googleapis/google-cloud-python/issues/7215) in which nested
+# transforms ({"a.c": "ServerTimestamp"}) next to nested values ({"a.b": 7})
+# incorrectly caused the fieldmask "a" to be set, which has the effect of wiping
+# out all data in "a" other than what was specified in the json-like input.
+
+# Instead, as this test specifies, transforms should not affect the fieldmask.
+
+description: "update: Nested transforms should not affect the field mask, even\nwhen there are other values that do. Transforms should only affect the\nDocumentTransform_FieldTransform list."
+update: <
+  doc_ref_path: "projects/projectID/databases/(default)/documents/C/d"
+  json_data: "{\"a.b\": 7, \"a.c\": \"ServerTimestamp\"}"
+  request: <
+    database: "projects/projectID/databases/(default)"
+    writes: <
+      update: <
+        name: "projects/projectID/databases/(default)/documents/C/d"
+        fields: <
+          key: "a"
+          value: <
+            map_value: <
+              fields: <
+                key: "b"
+                value: <
+                  integer_value: 7
+                >
+              >
+            >
+          >
+        >
+      >
+      update_mask: <
+        field_paths: "a.b"
+      >
+      current_document: <
+        exists: true
+      >
+    >
+    writes: <
+      transform: <
+        document: "projects/projectID/databases/(default)/documents/C/d"
+        field_transforms: <
+          field_path: "a.c"
+          set_to_server_value: REQUEST_TIME
+        >
+      >
+    >
+  >
+>