Repository: googleapis/google-cloud-python

Issue:
Firestore: Incorrect implementation of collection.add()
#### Environment details

Platforms: all

```
$ python --version
Python 2.7.17rc1
```

```
$ pip show google-cloud-firestore
Name: google-cloud-firestore
Version: 1.5.0
Summary: Google Cloud Firestore API client library
Home-page: https://github.com/GoogleCloudPlatform/google-cloud-python
Author: Google LLC
Author-email: googleapis-packages@google.com
License: Apache 2.0
Location: /usr/local/google/home/samstern/.local/lib/python2.7/site-packages
Requires: google-api-core, google-cloud-core, pytz
Required-by: firebase-admin
```

#### Steps to reproduce

This was originally reported to me here, and I reproduced it:
https://github.com/firebase/firebase-functions/issues/574


Python code:
```py
import firebase_admin
from firebase_admin import credentials
firebase_admin.initialize_app()
from google.cloud import firestore
db = firestore.Client()
print 'Adding'
db.collection('messages').add({'message':'Hello from Python'})
print 'Added'
```

Cloud Functions:
```js
const functions = require("firebase-functions");

exports.onNewMessageLength = functions.firestore
    .document("messages/{messageId}")
    .onCreate((snap, context) => {
        console.log("onCreate Data: ", JSON.stringify(snap.data()));
        return 0;
    });

exports.onUpdatedMessageLength = functions.firestore
    .document("messages/{messageId}")
    .onUpdate((change, context) => {
        console.log("onUpdate Data: ", JSON.stringify(change.after.data()));
        return 0;
    });
```

Running this in the Firebase emulators I see:
```
$ firebase emulators:exec "GCLOUD_PROJECT='fir-dumpster' python py/main.py && sleep 5"
⚠  Your requested "node" version "8" doesn't match your global version "10"
✔  functions: Emulator started at http://localhost:5001
i  firestore: Serving ALL traffic (including WebChannel) on http://localhost:8080
⚠  firestore: Support for WebChannel on a separate port (8081) is DEPRECATED and will go away soon. Please use port above instead.
i  firestore: Emulator logging to firestore-debug.log
✔  firestore: Emulator started at http://localhost:8080
i  firestore: For testing set FIRESTORE_EMULATOR_HOST=localhost:8080
i  functions: Watching "/tmp/tmp.OWPbChnnJ2/functions" for Cloud Functions...
✔  functions[onNewMessageLength]: firestore function initialized.
✔  functions[onUpdatedMessageLength]: firestore function initialized.
i  Running script: GCLOUD_PROJECT='fir-dumpster' python py/main.py && sleep 5
/usr/local/google/home/samstern/.local/lib/python2.7/site-packages/google/auth/_default.py:66: UserWarning: Your application has authenticated using end user credentials from Google Cloud SDK. We recommend that most server applications use service accounts instead. If your application continues to use end user credentials from Cloud SDK, you might receive a "quota exceeded" or "API not enabled" error. For more information about service accounts, see https://cloud.google.com/docs/authentication/.
  warnings.warn(_CLOUD_SDK_CREDENTIALS_WARNING)
Adding
Added
i  functions: Beginning execution of "onNewMessageLength"
>  onCreate Data:  {}
i  functions: Finished "onNewMessageLength" in ~1s
i  functions: Beginning execution of "onUpdatedMessageLength"
>  onUpdate Data:  {"message":{"type":"Buffer","data":[72,101,108,108,111,32,102,114,111,109,32,80,121,116,104,111,110]}}
i  functions: Finished "onUpdatedMessageLength" in ~1s
✔  Script exited successfully (code 0)
i  Shutting down emulators.
i  Stopping functions emulator
i  Stopping firestore emulator
```

What I should see is only the `onCreate` function firing and it should have the full data.  The implementation here is not correct:
https://github.com/googleapis/google-cloud-python/blob/master/firestore/google/cloud/firestore_v1/collection.py#L159-L180




Discussion:
cc @BenWhitehead 
@tseaver Can you take a look at this when you have a chance?
/cc @crwilcox 
Can we get this nailed down to something that doesn't involve `firesbase`, emulator output, or running anything on Cloud Functions?  E.g., if I run the Python script from above, what API calls do you expect it to make to the "real" backend API?  What state should be present?  Stripping down to stuff using only `google-cloud-firestore`:

```python
>>> import os
>>> from google.cloud.firestore_v1 import Client
>>> from google.oauth2 import service_account
>>> creds_file = os.environ['FIRESTORE_APPLICATION_CREDENTIALS']
>>> creds = service_account.Credentials.from_service_account_file(creds_file)
>>> client = Client(project=creds.project_id, credentials=creds)
>>> collection = client.collection('gcp-9629')
>>> doc_data = {'message': 'Reproduce https://github.com/googleapis/google-cloud-python/issues/9629'}
>>> timestamp, doc_ref = collection.add(doc_data)
>>> timestamp
seconds: 1573147518
nanos: 498621000

>>> doc_ref.path
'gcp-9629/V6XuLJ4fNIZizOcVYrxn'
>>> snapshot = doc_ref.get()
>>> snapshot.to_dict() == doc_data
True
>>> refs = list(collection.list_documents())
>>> refs[0] == doc_ref
True
```

That all looks correct to me, in terms of state visible from the outside:  we don't pass a document ID to `collection.add`, which means that it calls the `CreateDocument` API to get the server-generated document ID, and then calls the `Commit` API passing through the write PB to set the data.

Is the point of the issue to pass the document data along in the original `CreateDocument` API request, and avoid the `Commit` call?
@tseaver `add()` is short for "locally generate a random document ID and then call `set()`"

The Node.js implementation is pretty clear:
https://github.com/googleapis/nodejs-firestore/blob/master/dev/src/reference.ts#L2129-L2133

So yes I believe what you want to do is pass the data into `_firestore_api.create_document()` and avoid any further API calls.
Oh and in Node here's the implementation of `autoId()` for locally generating a document ID:
https://github.com/googleapis/nodejs-firestore/blob/master/dev/src/util.ts#L47-L54
@samtstern The call to `doc_ref.set()` is what triggers the `Commit` API call.  We can't pass the document data through to `_firestore_api.create_document`, because it might contain server transforms, which have to be processed in `Commit` (see issue #6862, fixed in PR #7072).

If generating a random document ID locally is OK / preferred, then we can avoid the call to the `CreateDocument` API, and just call `doc_ref.create`, which will invoke `Commit`.  We have a [comparable `_auto_id` function](https://github.com/googleapis/google-cloud-python/blob/79bdd03817879e495cae3534cbc168c3bf517ff2/firestore/google/cloud/firestore_v1/collection.py#L467-L474) already, which we [use in `Collection.document`](https://github.com/googleapis/google-cloud-python/blob/79bdd03817879e495cae3534cbc168c3bf517ff2/firestore/google/cloud/firestore_v1/collection.py#L94-L111).
@tseaver that makes sense.  Yes generating the random ID locally is preferred.

Resolving patch:
diff --git a/firestore/google/cloud/firestore_v1/collection.py b/firestore/google/cloud/firestore_v1/collection.py
--- a/firestore/google/cloud/firestore_v1/collection.py
+++ b/firestore/google/cloud/firestore_v1/collection.py
@@ -20,7 +20,6 @@
 
 from google.cloud.firestore_v1 import _helpers
 from google.cloud.firestore_v1 import query as query_mod
-from google.cloud.firestore_v1.proto import document_pb2
 from google.cloud.firestore_v1.watch import Watch
 from google.cloud.firestore_v1 import document
 
@@ -157,27 +156,11 @@ def add(self, document_data, document_id=None):
                 and the document already exists.
         """
         if document_id is None:
-            parent_path, expected_prefix = self._parent_info()
-
-            document_pb = document_pb2.Document()
-
-            created_document_pb = self._client._firestore_api.create_document(
-                parent_path,
-                collection_id=self.id,
-                document_id=None,
-                document=document_pb,
-                mask=None,
-                metadata=self._client._rpc_metadata,
-            )
+            document_id = _auto_id()
 
-            new_document_id = _helpers.get_doc_id(created_document_pb, expected_prefix)
-            document_ref = self.document(new_document_id)
-            set_result = document_ref.set(document_data)
-            return set_result.update_time, document_ref
-        else:
-            document_ref = self.document(document_id)
-            write_result = document_ref.create(document_data)
-            return write_result.update_time, document_ref
+        document_ref = self.document(document_id)
+        write_result = document_ref.create(document_data)
+        return write_result.update_time, document_ref
 
     def list_documents(self, page_size=None):
         """List all subdocuments of the current collection.


Test patch:
diff --git a/firestore/tests/unit/v1/test_collection.py b/firestore/tests/unit/v1/test_collection.py
--- a/firestore/tests/unit/v1/test_collection.py
+++ b/firestore/tests/unit/v1/test_collection.py
@@ -12,7 +12,6 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-import datetime
 import types
 import unittest
 
@@ -193,7 +192,7 @@ def test_add_auto_assigned(self):
         from google.cloud.firestore_v1.proto import document_pb2
         from google.cloud.firestore_v1.document import DocumentReference
         from google.cloud.firestore_v1 import SERVER_TIMESTAMP
-        from google.cloud.firestore_v1._helpers import pbs_for_set_no_merge
+        from google.cloud.firestore_v1._helpers import pbs_for_create
 
         # Create a minimal fake GAPIC add attach it to a real client.
         firestore_api = mock.Mock(spec=["create_document", "commit"])
@@ -214,42 +213,32 @@ def test_add_auto_assigned(self):
         # Actually make a collection.
         collection = self._make_one("grand-parent", "parent", "child", client=client)
 
-        # Add a dummy response for the fake GAPIC.
-        parent_path = collection.parent._document_path
-        auto_assigned_id = "cheezburger"
-        name = "{}/{}/{}".format(parent_path, collection.id, auto_assigned_id)
-        create_doc_response = document_pb2.Document(name=name)
-        create_doc_response.update_time.FromDatetime(datetime.datetime.utcnow())
-        firestore_api.create_document.return_value = create_doc_response
-
         # Actually call add() on our collection; include a transform to make
         # sure transforms during adds work.
         document_data = {"been": "here", "now": SERVER_TIMESTAMP}
-        update_time, document_ref = collection.add(document_data)
+
+        patch = mock.patch("google.cloud.firestore_v1.collection._auto_id")
+        random_doc_id = "DEADBEEF"
+        with patch as patched:
+            patched.return_value = random_doc_id
+            update_time, document_ref = collection.add(document_data)
 
         # Verify the response and the mocks.
         self.assertIs(update_time, mock.sentinel.update_time)
         self.assertIsInstance(document_ref, DocumentReference)
         self.assertIs(document_ref._client, client)
-        expected_path = collection._path + (auto_assigned_id,)
+        expected_path = collection._path + (random_doc_id,)
         self.assertEqual(document_ref._path, expected_path)
 
-        expected_document_pb = document_pb2.Document()
-        firestore_api.create_document.assert_called_once_with(
-            parent_path,
-            collection_id=collection.id,
-            document_id=None,
-            document=expected_document_pb,
-            mask=None,
-            metadata=client._rpc_metadata,
-        )
-        write_pbs = pbs_for_set_no_merge(document_ref._document_path, document_data)
+        write_pbs = pbs_for_create(document_ref._document_path, document_data)
         firestore_api.commit.assert_called_once_with(
             client._database_string,
             write_pbs,
             transaction=None,
             metadata=client._rpc_metadata,
         )
+        # Since we generate the ID locally, we don't call 'create_document'.
+        firestore_api.create_document.assert_not_called()
 
     @staticmethod
     def _write_pb_for_create(document_path, document_data):