Repository: googleapis/google-cloud-python

Issue:
Firestore: Using offset greater than 1000 on a query doesn't work
#### Environment details

macOS Mojave 10.14.1
Python 3.6
google-cloud-firestore==0.30.0

#### Steps to reproduce

  1. Create a query to get a large collection (over 1000 documents)
  2. Add an offset greater than 1000 to the query
  3. Get the query and iterate over the documents
  4. You should get an exception about having no documents

#### Code example

```python
from google.cloud import firestore

client = firestore.Client()

documents = client.collection('Collection').order_by('field').offset(1001).get()

for document in documents:
    pass
```

#### Stack trace
```
ValueError: Unexpected server response. All responses other than the first must contain a document. The response at index 1 was
read_time {
  seconds: 1544736718
  nanos: 109409000
}
skipped_results: 1
.
```


Discussion:
Can you please show the whole traceback?  Also, are you sure there are at least 1000 documents in the collection?
Sure.

```
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/Source/....../scripts/.......py", line 21, in <module>
    for document in documents:
  File "/Source/......../venv/lib/python3.6/site-packages/google/cloud/firestore_v1beta1/query.py", line 599, in get
    raise ValueError(msg)
ValueError: Unexpected server response. All responses other than the first must contain a document. The response at index 1 was
read_time {
  seconds: 1544741917
  nanos: 761441000
}
skipped_results: 1
.
```

I am certain there are more than 1000 documents as making the same query with an offset of 1000, returns more documents.
There is no indication in the [`StructuredQuery` API docs](https://cloud.google.com/firestore/docs/reference/rpc/google.firestore.v1beta1#google.firestore.v1beta1.StructuredQuery) of any upper bound on `limit`, and the `google-cloud-firestore` code doesn't do anything particular with the value beyond copying it into that message.

@shollyman Can you shed light on why the server would be returning a `RunQueryResponse` here with no `document` set?  If it is expected, what is the expected behavior within the client library?
on: 0.31.0

my stack trace is when I do `.offset(1001)`
```
  File "/usr/local/lib/python3.7/site-packages/google/cloud/firestore_v1beta1/query.py", line 758, in get
    raise ValueError(msg)
ValueError: Unexpected server response. All responses other than the first must contain a document. The response at index 1 was
read_time {
  seconds: 1547577043
  nanos: 738488000
}
skipped_results: 200
```
same issue, abandoning firestore for the moment as we can't get data out
```python
page_size = 1000
cursor = 0
limit = 100000
data = []

t_collection = client.collection(collection)
while True:
    t_query = t_collection.offset(cursor).limit(page_size)
    docs = [x.to_dict() for x in list(t_query.get())]
    if(len(docs)) == 0 or cursor >= limit:
        break
    else:
        data = data + docs
        print(cursor, len(docs), len(data))
        cursor += len(docs)
    
print('were done', len(data))
```
has following response after getting ~2k documents out
```
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-9-e21161698f76> in <module>
      7 while True:
      8     t_query = t_collection.offset(cursor).limit(page_size)
----> 9     docs = [x.to_dict() for x in list(t_query.get())]
     10     if(len(docs)) == 0 or cursor >= limit:
     11         break

~/.virtualenvs/broadleaf/lib/python3.7/site-packages/google/cloud/firestore_v1beta1/query.py in get(self, transaction)
    756                 if index != 0:
    757                     msg = _EMPTY_DOC_TEMPLATE.format(index, response_pb)
--> 758                     raise ValueError(msg)
    759                 empty_stream = skipped_results == 0
    760             else:

ValueError: Unexpected server response. All responses other than the first must contain a document. The response at index 1 was
read_time {
  seconds: 1548254968
  nanos: 205102000
}
skipped_results: 1000
.
```

Hmm, looking at the [API docs for `RunQueryResponse`](https://cloud.google.com/firestore/docs/reference/rpc/google.firestore.v1beta1#runqueryresponse), I note that the `document` field description says:

> A query result. Not set when reporting partial progress.

Response fields:

- The `transaction` field can only be set on the first response in the stream.
- `read_time` doesn't seem sensible if there is no document.
- `skipped_results`: I *guess* the back end could send a document-less response with this field set as a "still skipping" notification?
Looking at `Query.get`, which [loops over the stream of `RunQueryResponse` messages](https://github.com/googleapis/google-cloud-python/blob/4ccf5682e5674a09a52e5fbe1e4035b005d4b92c/firestore/google/cloud/firestore_v1beta1/query.py#L744-L761), ISTM that we could just *not* raise the error for the empty document.  In fact, the whole thing could be simplified to the following, given that we don't return the `skipped_results` anywhere:

```python
        for response_pb in response_iterator:
            snapshot, _ = _query_response_to_snapshot(
                response_pb, self._parent, expected_prefix
            )
            if snapshot is not None:
                yield snapshot
```

I don't think raising exceptions about potential error cases from the back-end (non-empty response after empty first response, empty response after non-empty response) is useful here, given that we *know* the back end can return the empty response beyond the first one.

Resolving patch:
diff --git a/firestore/google/cloud/firestore_v1beta1/query.py b/firestore/google/cloud/firestore_v1beta1/query.py
--- a/firestore/google/cloud/firestore_v1beta1/query.py
+++ b/firestore/google/cloud/firestore_v1beta1/query.py
@@ -63,10 +63,6 @@
     "come from fields set in ``order_by()``."
 )
 _MISMATCH_CURSOR_W_ORDER_BY = "The cursor {!r} does not match the order fields {!r}."
-_EMPTY_DOC_TEMPLATE = (
-    "Unexpected server response. All responses other than the first must "
-    "contain a document. The response at index {} was\n{}."
-)
 
 
 class Query(object):
@@ -725,12 +721,6 @@ def get(self, transaction=None):
         Yields:
             ~.firestore_v1beta1.document.DocumentSnapshot: The next
             document that fulfills the query.
-
-        Raises:
-            ValueError: If the first response in the stream is empty, but
-                then more responses follow.
-            ValueError: If a response other than the first does not contain
-                a document.
         """
         parent_path, expected_prefix = self._parent._parent_info()
         response_iterator = self._client._firestore_api.run_query(
@@ -740,24 +730,11 @@ def get(self, transaction=None):
             metadata=self._client._rpc_metadata,
         )
 
-        empty_stream = False
-        for index, response_pb in enumerate(response_iterator):
-            if empty_stream:
-                raise ValueError(
-                    "First response in stream was empty",
-                    "Received second response",
-                    response_pb,
-                )
-
-            snapshot, skipped_results = _query_response_to_snapshot(
-                response_pb, self._parent, expected_prefix
+        for response in response_iterator:
+            snapshot = _query_response_to_snapshot(
+                response, self._parent, expected_prefix
             )
-            if snapshot is None:
-                if index != 0:
-                    msg = _EMPTY_DOC_TEMPLATE.format(index, response_pb)
-                    raise ValueError(msg)
-                empty_stream = skipped_results == 0
-            else:
+            if snapshot is not None:
                 yield snapshot
 
     def on_snapshot(self, callback):
@@ -964,13 +941,12 @@ def _query_response_to_snapshot(response_pb, collection, expected_prefix):
             directly from ``collection`` via :meth:`_parent_info`.
 
     Returns:
-        Tuple[Optional[~.firestore.document.DocumentSnapshot], int]: A
-        snapshot of the data returned in the query and the number of skipped
-        results. If ``response_pb.document`` is not set, the snapshot will be
-        :data:`None`.
+        Optional[~.firestore.document.DocumentSnapshot]: A
+        snapshot of the data returned in the query. If ``response_pb.document``
+        is not set, the snapshot will be :data:`None`.
     """
     if not response_pb.HasField("document"):
-        return None, response_pb.skipped_results
+        return None
 
     document_id = _helpers.get_doc_id(response_pb.document, expected_prefix)
     reference = collection.document(document_id)
@@ -983,4 +959,4 @@ def _query_response_to_snapshot(response_pb, collection, expected_prefix):
         create_time=response_pb.document.create_time,
         update_time=response_pb.document.update_time,
     )
-    return snapshot, response_pb.skipped_results
+    return snapshot


Test patch:
diff --git a/firestore/tests/unit/test_query.py b/firestore/tests/unit/test_query.py
--- a/firestore/tests/unit/test_query.py
+++ b/firestore/tests/unit/test_query.py
@@ -1137,13 +1137,7 @@ def test_get_second_response_in_empty_stream(self):
 
         get_response = query.get()
         self.assertIsInstance(get_response, types.GeneratorType)
-        with self.assertRaises(ValueError) as exc_info:
-            list(get_response)
-
-        exc_args = exc_info.exception.args
-        self.assertEqual(len(exc_args), 3)
-        self.assertIs(exc_args[2], empty_response2)
-        self.assertIsNot(empty_response1, empty_response2)
+        self.assertEqual(list(get_response), [])
 
         # Verify the mock call.
         parent_path, _ = parent._parent_info()
@@ -1193,8 +1187,6 @@ def test_get_with_skipped_results(self):
         )
 
     def test_get_empty_after_first_response(self):
-        from google.cloud.firestore_v1beta1.query import _EMPTY_DOC_TEMPLATE
-
         # Create a minimal fake GAPIC.
         firestore_api = mock.Mock(spec=["run_query"])
 
@@ -1217,13 +1209,11 @@ def test_get_empty_after_first_response(self):
         query = self._make_one(parent)
         get_response = query.get()
         self.assertIsInstance(get_response, types.GeneratorType)
-        with self.assertRaises(ValueError) as exc_info:
-            list(get_response)
-
-        exc_args = exc_info.exception.args
-        self.assertEqual(len(exc_args), 1)
-        msg = _EMPTY_DOC_TEMPLATE.format(1, response_pb2)
-        self.assertEqual(exc_args[0], msg)
+        returned = list(get_response)
+        self.assertEqual(len(returned), 1)
+        snapshot = returned[0]
+        self.assertEqual(snapshot.reference._path, ("charles", "bark"))
+        self.assertEqual(snapshot.to_dict(), data)
 
         # Verify the mock call.
         parent_path, _ = parent._parent_info()
@@ -1468,16 +1458,14 @@ def _call_fut(response_pb, collection, expected_prefix):
 
     def test_empty(self):
         response_pb = _make_query_response()
-        snapshot, skipped_results = self._call_fut(response_pb, None, None)
+        snapshot = self._call_fut(response_pb, None, None)
         self.assertIsNone(snapshot)
-        self.assertEqual(skipped_results, 0)
 
     def test_after_offset(self):
         skipped_results = 410
         response_pb = _make_query_response(skipped_results=skipped_results)
-        snapshot, skipped_results = self._call_fut(response_pb, None, None)
+        snapshot = self._call_fut(response_pb, None, None)
         self.assertIsNone(snapshot)
-        self.assertEqual(skipped_results, skipped_results)
 
     def test_response(self):
         from google.cloud.firestore_v1beta1.document import DocumentSnapshot
@@ -1492,10 +1480,7 @@ def test_response(self):
         data = {"a": 901, "b": True}
         response_pb = _make_query_response(name=name, data=data)
 
-        snapshot, skipped_results = self._call_fut(
-            response_pb, collection, expected_prefix
-        )
-        self.assertEqual(skipped_results, 0)
+        snapshot = self._call_fut(response_pb, collection, expected_prefix)
         self.assertIsInstance(snapshot, DocumentSnapshot)
         expected_path = collection._path + (doc_id,)
         self.assertEqual(snapshot.reference._path, expected_path)