Repository: googleapis/google-cloud-python

Issue:
BigTable: google-cloud-bigtable 0.30.0 throws InvalidChunk error while iterating through a large collection
Python version: 3.6.3
google-cloud-bigtable 0.30.0

After switching to 0.30.0 and adapting my code to use the newly integrated iter function of PartialRowsData, I can't iterate through a large collection without the InvalidChunk error being thrown.

Sample code:
```
rows = table.read_rows()
for r in row:
...
```



Discussion:
@xEcEz Can you please provide the full traceback?  Also, can you guesstimate the size of the table (number of rows, column families, etc.)?
Stacktrace:
```
at _validate_cell_data_new_row (/usr/lib/python3.6/site-packages/google/cloud/bigtable/row_data.py:517)
at _validate_cell_data (/usr/lib/python3.6/site-packages/google/cloud/bigtable/row_data.py:505)
at __iter__ (/usr/lib/python3.6/site-packages/google/cloud/bigtable/row_data.py:474)
at readAll (/scripts/bigtable_manager.py:106)
at call (/usr/lib/python3.6/site-packages/tenacity/__init__.py:361)
at __get_result (/usr/lib/python3.6/concurrent/futures/_base.py:384)
at result (/usr/lib/python3.6/concurrent/futures/_base.py:425)
at reraise (/usr/lib/python3.6/site-packages/tenacity/__init__.py:167)
at iter (/usr/lib/python3.6/site-packages/tenacity/__init__.py:331)
at call (/usr/lib/python3.6/site-packages/tenacity/__init__.py:358)
at wrapped_f (/usr/lib/python3.6/site-packages/tenacity/__init__.py:292)
at retryHandler (/scripts/bigtable_manager.py:149)
at getAllRanksTiers (/scripts/bigtable_manager.py:268)
at run (/scripts/queue_processor_update.py:68)
at _bootstrap (/usr/lib/python3.6/multiprocessing/process.py:258)
```

Table size: approx. 2.5M rows, 1 column family containing small JSON docs of 2 fields.
@AVaksman, can you please take a look at this?
I've been looking at this.  `row_data.py:517` indicates that the client sees row_keys that are out of order. 
 v0.29.0 and before didn't have that validation.

We can relax the validation in the short term, but it really ought to be fixed.  The service doesn't send rows out of sequence, so my guess is that there's some sort of unexpected retry that happens that restarts the request from the beginning.  

I'm going to run a long test over the week end to see if I can reproduce the problem.  I'm also going to do some refactoring of [`row_data.py`](https://github.com/GoogleCloudPlatform/google-cloud-python/blob/master/bigtable/google/cloud/bigtable/row_data.py) so that I can better rationalize about it a bit more.
I've had this issue show up and stably happen at a single row. I'm guessing that was a tablet boundary, which may give some clues as to the origin. (Unfortunately, that table has changed a lot since then, so I can't recheck and see if this was a time-transient error -- suggesting something retry-associated -- or not)
I'm going to remove this check for now.  I wonder if this issue relates to the fact that Cloud Bigtable has binary data, and `<=` checks may not be accurate due to some kind of encoding issue, or difference between Python and Cloud Bigtable's implementation.
Ugh.  `'C' <= 'f'`, which is true for python strings, but Bigtable's lexicographical ordering is different and is ordered based on byte values.  We'll need a Python equivalent of this Java class: [ByteStringComparator](https://github.com/GoogleCloudPlatform/cloud-bigtable-client/blob/edc800c1c7f188f4cc70c5746c80cb67de0d4df3/bigtable-client-core-parent/bigtable-client-core/src/main/java/com/google/cloud/bigtable/util/ByteStringComparator.java).
Not necessarily for the way?

On Fri, Sep 7, 2018, 12:51 Solomon Duskis <notifications@github.com> wrote:

> Ugh. 'C' <= 'f', which is true for non-strings, but not necessarily for
> the way . We'll need a Python equivalent of this Java class:
> ByteStringComparator
> <https://github.com/GoogleCloudPlatform/cloud-bigtable-client/blob/edc800c1c7f188f4cc70c5746c80cb67de0d4df3/bigtable-client-core-parent/bigtable-client-core/src/main/java/com/google/cloud/bigtable/util/ByteStringComparator.java>
> .
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/GoogleCloudPlatform/google-cloud-python/issues/5876#issuecomment-419547976>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AdpQl9J1hMuYvVcjVU-P0yQA89VYnzLQks5uYs45gaJpZM4WVtNX>
> .
>

@zunger-humu, I updated the comment...
ok.  'C' is actually less than 'f' in ascii, so '<=' ought to work correctly for ascii.
I raised an internal issue to see if there's anything else going on.
More data: I just (accidentally) had some code running on 0.30 instead of
0.29, where it was doing a read_rows over about 2700 rows of data. If it
just did the read_rows, everything was fine, but if the read_rows was
intermixed with a mutate operation (batched, but getting several batches
off before things went wrong) the read would stop after 274 rows, with no
error or sign that anything was less than perfect. The row keys in this
case were all 16 byte hex expressed as ASCII, so they should be super-nice
from a code perspective.

I have no idea what's going on here but 0.30 is very scary. If one of our
partner experience people hadn't noticed that a certain output file had way
fewer entries than it was supposed to, we would have just messed up some
user data...

On Fri, Sep 7, 2018 at 2:41 PM Solomon Duskis <notifications@github.com>
wrote:

> I raised an internal issue to see if there's anything else going on.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/GoogleCloudPlatform/google-cloud-python/issues/5876#issuecomment-419573279>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AdpQl6qszJ5ovl-RHEx19PqoKtlOjoRvks5uYugKgaJpZM4WVtNX>
> .
>


Resolving patch:
diff --git a/bigtable/google/cloud/bigtable/row_data.py b/bigtable/google/cloud/bigtable/row_data.py
--- a/bigtable/google/cloud/bigtable/row_data.py
+++ b/bigtable/google/cloud/bigtable/row_data.py
@@ -406,15 +406,16 @@ def _create_retry_request(self):
         req_manager = _ReadRowsRequestManager(self.request,
                                               self.last_scanned_row_key,
                                               self._counter)
-        self.request = req_manager.build_updated_request()
+        return req_manager.build_updated_request()
 
     def _on_error(self, exc):
         """Helper for :meth:`__iter__`."""
         # restart the read scan from AFTER the last successfully read row
+        retry_request = self.request
         if self.last_scanned_row_key:
-            self._create_retry_request()
+            retry_request = self._create_retry_request()
 
-        self.response_iterator = self.read_method(self.request)
+        self.response_iterator = self.read_method(retry_request)
 
     def _read_next(self):
         """Helper for :meth:`__iter__`."""
@@ -604,11 +605,18 @@ def build_updated_request(self):
             r_kwargs['rows_limit'] = max(1, self.message.rows_limit -
                                          self.rows_read_so_far)
 
-        row_keys = self._filter_rows_keys()
-        row_ranges = self._filter_row_ranges()
-        r_kwargs['rows'] = data_v2_pb2.RowSet(row_keys=row_keys,
-                                              row_ranges=row_ranges)
-
+        # if neither RowSet.row_keys nor RowSet.row_ranges currently exist,
+        # add row_range that starts with last_scanned_key as start_key_open
+        # to request only rows that have not been returned yet
+        if not self.message.HasField('rows'):
+            row_range = data_v2_pb2.RowRange(
+                start_key_open=self.last_scanned_key)
+            r_kwargs['rows'] = data_v2_pb2.RowSet(row_ranges=[row_range])
+        else:
+            row_keys = self._filter_rows_keys()
+            row_ranges = self._filter_row_ranges()
+            r_kwargs['rows'] = data_v2_pb2.RowSet(row_keys=row_keys,
+                                                  row_ranges=row_ranges)
         return data_messages_v2_pb2.ReadRowsRequest(**r_kwargs)
 
     def _filter_rows_keys(self):
@@ -621,22 +629,29 @@ def _filter_row_ranges(self):
         new_row_ranges = []
 
         for row_range in self.message.rows.row_ranges:
-            if((row_range.end_key_open and
-                self._key_already_read(row_range.end_key_open)) or
-                (row_range.end_key_closed and
-                 self._key_already_read(row_range.end_key_closed))):
-                    continue
-
-            if ((row_range.start_key_open and
-                self._key_already_read(row_range.start_key_open)) or
-                (row_range.start_key_closed and
-                 self._key_already_read(row_range.start_key_closed))):
-                row_range.start_key_closed = _to_bytes("")
-                row_range.start_key_open = self.last_scanned_key
-
-                new_row_ranges.append(row_range)
-            else:
-                new_row_ranges.append(row_range)
+            # if current end_key (open or closed) is set, return its value,
+            # if not, set to empty string ('').
+            # NOTE: Empty string in end_key means "end of table"
+            end_key = self._end_key_set(row_range)
+            # if end_key is already read, skip to the next row_range
+            if(end_key and self._key_already_read(end_key)):
+                continue
+
+            # if current start_key (open or closed) is set, return its value,
+            # if not, then set to empty string ('')
+            # NOTE: Empty string in start_key means "beginning of table"
+            start_key = self._start_key_set(row_range)
+
+            # if start_key was already read or doesn't exist,
+            # create a row_range with last_scanned_key as start_key_open
+            # to be passed to retry request
+            retry_row_range = row_range
+            if(self._key_already_read(start_key)):
+                retry_row_range = copy.deepcopy(row_range)
+                retry_row_range.start_key_closed = _to_bytes("")
+                retry_row_range.start_key_open = self.last_scanned_key
+
+            new_row_ranges.append(retry_row_range)
 
         return new_row_ranges
 
@@ -644,6 +659,16 @@ def _key_already_read(self, key):
         """ Helper for :meth:`_filter_row_ranges`"""
         return key <= self.last_scanned_key
 
+    @staticmethod
+    def _start_key_set(row_range):
+        """ Helper for :meth:`_filter_row_ranges`"""
+        return row_range.start_key_open or row_range.start_key_closed
+
+    @staticmethod
+    def _end_key_set(row_range):
+        """ Helper for :meth:`_filter_row_ranges`"""
+        return row_range.end_key_open or row_range.end_key_closed
+
 
 def _raise_if(predicate, *args):
     """Helper for validation methods."""


Test patch:
diff --git a/bigtable/tests/unit/test_row_data.py b/bigtable/tests/unit/test_row_data.py
--- a/bigtable/tests/unit/test_row_data.py
+++ b/bigtable/tests/unit/test_row_data.py
@@ -795,12 +795,12 @@ def test__filter_row_ranges_some_ranges_already_read(self):
                                               end_key_open=b"row_key49")
         exp_row_ranges = [exp_row_range1, exp_row_range2, exp_row_range3]
 
-        self.assertEqual(row_ranges, exp_row_ranges)
+        self.assertEqual(exp_row_ranges, row_ranges)
 
     def test_build_updated_request(self):
         from google.cloud.bigtable.row_filters import RowSampleFilter
         row_filter = RowSampleFilter(0.33)
-        last_scanned_key = b"row_key14"
+        last_scanned_key = b"row_key25"
         request = _ReadRowsRequestPB(filter=row_filter.to_pb(),
                                      rows_limit=8,
                                      table_name=self.table_name)
@@ -813,9 +813,100 @@ def test_build_updated_request(self):
         expected_result = _ReadRowsRequestPB(table_name=self.table_name,
                                              filter=row_filter.to_pb(),
                                              rows_limit=6)
-        expected_result.rows.row_ranges.add(**self.row_range1.
-                                            get_range_kwargs())
+        expected_result.rows.row_ranges.add(
+            start_key_open=last_scanned_key,
+            end_key_open=self.row_range1.end_key
+        )
+
+        self.assertEqual(expected_result, result)
+
+    def test_build_updated_request_full_table(self):
+        last_scanned_key = b"row_key14"
+
+        request = _ReadRowsRequestPB(table_name=self.table_name)
+        request_manager = self._make_one(request, last_scanned_key, 2)
+
+        result = request_manager.build_updated_request()
+        expected_result = _ReadRowsRequestPB(table_name=self.table_name,
+                                             filter={})
+        expected_result.rows.row_ranges.add(start_key_open=last_scanned_key)
+        self.assertEqual(expected_result, result)
+
+    def test_build_updated_request_no_start_key(self):
+        from google.cloud.bigtable.row_filters import RowSampleFilter
+        row_filter = RowSampleFilter(0.33)
+        last_scanned_key = b"row_key25"
+        request = _ReadRowsRequestPB(filter=row_filter.to_pb(),
+                                     rows_limit=8,
+                                     table_name=self.table_name)
+        request.rows.row_ranges.add(end_key_open=b"row_key29")
+
+        request_manager = self._make_one(request, last_scanned_key, 2)
+
+        result = request_manager.build_updated_request()
+
+        expected_result = _ReadRowsRequestPB(table_name=self.table_name,
+                                             filter=row_filter.to_pb(),
+                                             rows_limit=6)
+        expected_result.rows.row_ranges.add(start_key_open=last_scanned_key,
+                                            end_key_open=b"row_key29")
+
+        self.assertEqual(expected_result, result)
+
+    def test_build_updated_request_no_end_key(self):
+        from google.cloud.bigtable.row_filters import RowSampleFilter
+        row_filter = RowSampleFilter(0.33)
+        last_scanned_key = b"row_key25"
+        request = _ReadRowsRequestPB(filter=row_filter.to_pb(),
+                                     rows_limit=8,
+                                     table_name=self.table_name)
+        request.rows.row_ranges.add(start_key_closed=b"row_key20")
+
+        request_manager = self._make_one(request, last_scanned_key, 2)
+
+        result = request_manager.build_updated_request()
+
+        expected_result = _ReadRowsRequestPB(table_name=self.table_name,
+                                             filter=row_filter.to_pb(),
+                                             rows_limit=6)
+        expected_result.rows.row_ranges.add(start_key_open=last_scanned_key)
+
+        self.assertEqual(expected_result, result)
 
+    def test_build_updated_request_rows(self):
+        from google.cloud.bigtable.row_filters import RowSampleFilter
+        row_filter = RowSampleFilter(0.33)
+        last_scanned_key = b"row_key4"
+        request = _ReadRowsRequestPB(filter=row_filter.to_pb(),
+                                     rows_limit=5,
+                                     table_name=self.table_name)
+        request.rows.row_keys.extend([b'row_key1', b'row_key2',
+                                      b'row_key4', b'row_key5',
+                                      b'row_key7', b'row_key9'])
+
+        request_manager = self._make_one(request, last_scanned_key, 3)
+
+        result = request_manager.build_updated_request()
+
+        expected_result = _ReadRowsRequestPB(table_name=self.table_name,
+                                             filter=row_filter.to_pb(),
+                                             rows_limit=2)
+        expected_result.rows.row_keys.extend([b'row_key5', b'row_key7',
+                                              b'row_key9'])
+
+        self.assertEqual(expected_result, result)
+
+    def test_build_updated_request_rows_limit(self):
+        last_scanned_key = b"row_key14"
+
+        request = _ReadRowsRequestPB(table_name=self.table_name, rows_limit=10)
+        request_manager = self._make_one(request, last_scanned_key, 2)
+
+        result = request_manager.build_updated_request()
+        expected_result = _ReadRowsRequestPB(table_name=self.table_name,
+                                             filter={},
+                                             rows_limit=8)
+        expected_result.rows.row_ranges.add(start_key_open=last_scanned_key)
         self.assertEqual(expected_result, result)
 
     def test__key_already_read(self):