Repository: googleapis/google-cloud-python

Issue:
Bigtable Python Client Max Row size is 4mb
On the big table documents, it says the max cell size is 100mb. However, when I try to read a row with a cell of size 10mb using the Bigtable Python client, I get the following error:

```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/google/cloud/happybase/table.py", line 190, in row
    row, filter_=filter_)
  File "/usr/local/lib/python2.7/dist-packages/google/cloud/bigtable/table.py", line 234, in read_row
    rows_data.consume_all()
  File "/usr/local/lib/python2.7/dist-packages/google/cloud/bigtable/row_data.py", line 323, in consume_all
    self.consume_next()
  File "/usr/local/lib/python2.7/dist-packages/google/cloud/bigtable/row_data.py", line 261, in consume_next
    response = six.next(self._response_iterator)
  File "/usr/local/lib/python2.7/dist-packages/grpc/_channel.py", line 344, in next
    return self._next()
  File "/usr/local/lib/python2.7/dist-packages/grpc/_channel.py", line 335, in _next
    raise self
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.INTERNAL, Max message size exceeded)>
```

This max size seems to be hard coded in the grpc library. Has anybody been able to read large rows using the Bigtable Python client? Any idea for workarounds or how I can set the max size? 


Discussion:
@nathanielmanistaatgoogle WDYT?
A few gRPC users have had this kind of problem and we've started a discussion in [`grpc/grpc.github.io` issue 371](https://github.com/grpc/grpc.github.io/issues/371).

For now you can cross your fingers and see what happens when you pass a channel options value like `options=(('grpc.max_message_length', 100 * 1024 * 1024),)` to [`grpc.insecure_channel`](http://www.grpc.io/grpc/python/grpc.html#grpc.insecure_channel) or [`grpc.secure_channel`](http://www.grpc.io/grpc/python/grpc.html#grpc.secure_channel) (know that `grpc.max_message_length` is broken up into `grpc.max_send_message_length` and `grpc.max_receive_message_length` in gRPC Python 1.1-and-later). It might work.

If it doesn't work, then you've hit some limit inside gRPC that isn't overridable and you'll have to break up the large message into a stream of smaller ones.
@nathanielmanistaatgoogle The "you" in "when you pass a channel options value" here is the library maintainers, yes? As a user, @brandon-white isn't directly creating any gRPC object, he is dealing with our Bigtable classes (i.e. the nouns) and calling the API via their instance methods (i.e. the verbs).
Thanks @nathanielmanistaatgoogle I'll try this but, ideally, I would prefer a solution where I do not edit the Bigtable Python client. 

For the Google Bigtable Client folks, have you encountered any use cases where the rows and cells are large? If so, how do you break up these large rows and cells into streams? Are large rows and cells supported through the Python Client? 
@mbrukman Have you run into this in other clients?
While there's a per-response limit in gRPC, it is possible to retrieve larger cell values via streaming responses and reconstructing them client-side. [`ReadRowsResponse`](https://cloud.google.com/bigtable/docs/reference/data/rpc/google.bigtable.v2#readrowsresponse) includes a list of [`CellChunk`](https://cloud.google.com/bigtable/docs/reference/data/rpc/google.bigtable.v2#cellchunk)s which are the pieces of cells that can be reconstructed to form the larger values.

/cc: @sduskis, @garye
Thank you @mbrukman ! I don't see these calls in the BigTable Python Client. Does this mean I need to write my own client if I want to consume these large cell values? Are there any plans to incorporate this into existing clients? 
The python client already deals with ReadRowResponse under the covers.  This seems like something we need to change in the service.
I'll take that back.  This seems like a client setting.  In the java client, we set the value on the netty channel (maxMessageSize) to be 256MB.  Is that a setting that the python client controls?
@sduskis I looked for that but didn't see any option to set that in the [Python client](https://googlecloudplatform.github.io/google-cloud-python/stable/_modules/google/cloud/bigtable/client.html#Client). So it seems like we need to add this max_message_size config to the BigTable Python Client? 
@dhermes, do you have any idea where grpc settings are set, and if max_message_size is a settable property?
I do not, though @nathanielmanistaatgoogle would be a good person to ask. I'm not 100% clear on if "grpc settings" are a global thing or a per-call setting? The [`options=` described above][1] is how we'd set gRPC metadata on a per-call basis.

[1]: https://github.com/GoogleCloudPlatform/google-cloud-python/issues/2880#issuecomment-268020287
It looks like the channel options get set in [`core/google/cloud/_helpers.py:489`](https://github.com/GoogleCloudPlatform/google-cloud-python/blob/d20f3bb7458915d2601b670b7f4df31bc9ead473/core/google/cloud/_helpers.py#L489). I don't see a way at the moment to pass options specifically for Bigtable.

I think [`grpc.max_receive_message_length`](https://github.com/grpc/grpc/blob/5904a952747ce2c95bd95c5a2b27dcdf41029a43/include/grpc/impl/codegen/grpc_types.h#L157) would be the option to set.

We might ultimately want to set the max send message length for Bigtable also. It looks like it is [currently unlimited](https://github.com/grpc/grpc/blob/5904a952747ce2c95bd95c5a2b27dcdf41029a43/include/grpc/impl/codegen/grpc_types.h#L260), but there is a *TODO* there to change that.
So do I need to make the PR to this or do any of the committers have bandwidth to take a look at this setting? What is the process for getting fixes like this out? 
Hello @brandon-white!
I think I have a patch for this, do you have any sample code that I can try with my branch?


@daspecster Thank you! I do not really have any custom code, I simply use the Bigtable Python API to query large cells. 

```
from google.cloud import bigtable
client = bigtable.Client(project=project_id, admin=True)
instance = client.instance(instance_id)
table = instance.table(table_id)
table.read_row(row_key)
```
It appears that passing the `('grpc.max_receive_message_length', 100 * 1024 * 1024)` in `options` doesn't work. :(

@mbrukman it appears to me that there is some handling of [chunks](https://github.com/GoogleCloudPlatform/google-cloud-python/blob/455eaf62653dd2ff6b4202a5392e059cf8eda5d0/bigtable/google/cloud/bigtable/row_data.py#L255) on the client side.

But in this case it appears to not get the chunks back yet.

```python
row_data = table.read_rows('large-row')
print(row_data.consume_next())
```

```bash
Traceback (most recent call last):
  File "/Documents/test_bigtable.py", line 14, in <module>
    print(row_data.consume_next())
  File "/Documents/test/.tox/py27/lib/python2.7/site-packages/google_cloud_bigtable-0.22.0-py2.7.egg/google/cloud/bigtable/row_data.py", line 261, in consume_next
    response = six.next(self._response_iterator)
  File "python_build/bdist.macosx-10.12-intel/egg/grpc/_channel.py", line 344, in next
  File "python_build/bdist.macosx-10.12-intel/egg/grpc/_channel.py", line 335, in _next
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.INTERNAL, Max message size exceeded)>
```

I'm missing how to get `ReadRowsRequest` to return a `ReadRowsResponse` where the chunks are 4mb or less?

`ReadRowsResponse`s are currently not chunked on the server sides, os there's no way to force it to happen.  That's an enhancement we wanted to allow via the API, but didn't implement yet.

I think we need some help from someone on the grpc team to solve this.
I wasn't sure based on the previous conversation. Thanks for clearing that up for me @sduskis!

Let me know if there's anything I can do to help. I'm not entirely sure who to ask about this?
I'll ping some folks who might know how to help.
Thanks all! So for now, it looks like this is a feature request which needs an owner? 
There's an [answer in this thread](https://github.com/GoogleCloudPlatform/google-cloud-python/issues/2880#issuecomment-268020287) to the question by the grpc python lead @nathanielmanistaatgoogle.  @gamorris's [answer](https://github.com/GoogleCloudPlatform/google-cloud-python/issues/2880#issuecomment-268020287) points to [the code that needs to be changed](https://github.com/GoogleCloudPlatform/google-cloud-python/blob/d20f3bb7458915d2601b670b7f4df31bc9ead473/core/google/cloud/_helpers.py#L489). 

@dhermes, are you the right person to change this?
@dhermes: the "you" in "when you pass a channel options value" is "the caller of the channel construction functions [`grpc.insecure_channel`](http://www.grpc.io/grpc/python/grpc.html#grpc.insecure_channel) and [`grpc.secure_channel`](http://www.grpc.io/grpc/python/grpc.html#grpc.secure_channel)".

Resolving patch:
diff --git a/bigtable/google/cloud/bigtable/client.py b/bigtable/google/cloud/bigtable/client.py
--- a/bigtable/google/cloud/bigtable/client.py
+++ b/bigtable/google/cloud/bigtable/client.py
@@ -65,6 +65,14 @@
 READ_ONLY_SCOPE = 'https://www.googleapis.com/auth/bigtable.data.readonly'
 """Scope for reading table data."""
 
+# NOTE: 'grpc.max_message_length' will no longer be recognized in
+#       grpcio 1.1 and later.
+_MAX_MSG_LENGTH_100MB = 100 * 1024 * 1024
+_GRPC_MAX_LENGTH_OPTIONS = (
+    ('grpc.max_message_length', _MAX_MSG_LENGTH_100MB),
+    ('grpc.max_receive_message_length', _MAX_MSG_LENGTH_100MB),
+)
+
 
 def _make_data_stub(client):
     """Creates gRPC stub to make requests to the Data API.
@@ -77,7 +85,8 @@ def _make_data_stub(client):
     """
     if client.emulator_host is None:
         return make_secure_stub(client.credentials, client.user_agent,
-                                bigtable_pb2.BigtableStub, DATA_API_HOST)
+                                bigtable_pb2.BigtableStub, DATA_API_HOST,
+                                extra_options=_GRPC_MAX_LENGTH_OPTIONS)
     else:
         return make_insecure_stub(bigtable_pb2.BigtableStub,
                                   client.emulator_host)
diff --git a/core/google/cloud/_helpers.py b/core/google/cloud/_helpers.py
--- a/core/google/cloud/_helpers.py
+++ b/core/google/cloud/_helpers.py
@@ -465,7 +465,7 @@ def _name_from_project_path(path, project, template):
     return match.group('name')
 
 
-def make_secure_channel(credentials, user_agent, host):
+def make_secure_channel(credentials, user_agent, host, extra_options=None):
     """Makes a secure channel for an RPC service.
 
     Uses / depends on gRPC.
@@ -480,14 +480,21 @@ def make_secure_channel(credentials, user_agent, host):
     :type host: str
     :param host: The host for the service.
 
+    :type extra_options: tuple
+    :param extra_options: (Optional) Extra gRPC options used when creating the
+                          channel.
+
     :rtype: :class:`grpc._channel.Channel`
     :returns: gRPC secure channel with credentials attached.
     """
     target = '%s:%d' % (host, http_client.HTTPS_PORT)
     http_request = google_auth_httplib2.Request(http=httplib2.Http())
-    options = (
-        ('grpc.primary_user_agent', user_agent),
-    )
+
+    user_agent_option = ('grpc.primary_user_agent', user_agent)
+    if extra_options is not None:
+        options = (user_agent_option,) + extra_options
+    else:
+        options = (user_agent_option,)
     return google.auth.transport.grpc.secure_authorized_channel(
         credentials,
         http_request,
@@ -495,7 +502,8 @@ def make_secure_channel(credentials, user_agent, host):
         options=options)
 
 
-def make_secure_stub(credentials, user_agent, stub_class, host):
+def make_secure_stub(credentials, user_agent, stub_class, host,
+                     extra_options=None):
     """Makes a secure stub for an RPC service.
 
     Uses / depends on gRPC.
@@ -513,10 +521,15 @@ def make_secure_stub(credentials, user_agent, stub_class, host):
     :type host: str
     :param host: The host for the service.
 
+    :type extra_options: tuple
+    :param extra_options: (Optional) Extra gRPC options passed when creating
+                          the channel.
+
     :rtype: object, instance of ``stub_class``
     :returns: The stub object used to make gRPC requests to a given API.
     """
-    channel = make_secure_channel(credentials, user_agent, host)
+    channel = make_secure_channel(credentials, user_agent, host,
+                                  extra_options=extra_options)
     return stub_class(channel)
 
 


Test patch:
diff --git a/bigtable/unit_tests/test_client.py b/bigtable/unit_tests/test_client.py
--- a/bigtable/unit_tests/test_client.py
+++ b/bigtable/unit_tests/test_client.py
@@ -46,13 +46,18 @@ def test_without_emulator(self):
         fake_stub = object()
         make_secure_stub_args = []
 
-        def mock_make_secure_stub(*args):
+        def mock_make_secure_stub(*args, **kwargs):
             make_secure_stub_args.append(args)
+            make_secure_stub_args.append(kwargs)
             return fake_stub
 
         with _Monkey(MUT, make_secure_stub=mock_make_secure_stub):
             result = self._call_fut(client)
 
+        extra_options = {'extra_options': (
+            ('grpc.max_message_length', 104857600),
+            ('grpc.max_receive_message_length', 104857600)
+        )}
         self.assertIs(result, fake_stub)
         self.assertEqual(make_secure_stub_args, [
             (
@@ -61,6 +66,7 @@ def mock_make_secure_stub(*args):
                 MUT.bigtable_pb2.BigtableStub,
                 MUT.DATA_API_HOST,
             ),
+            extra_options,
         ])
 
     def test_with_emulator(self):
diff --git a/core/unit_tests/test__helpers.py b/core/unit_tests/test__helpers.py
--- a/core/unit_tests/test__helpers.py
+++ b/core/unit_tests/test__helpers.py
@@ -644,6 +644,33 @@ def test_it(self):
         secure_authorized_channel.assert_called_once_with(
             credentials, mock.ANY, expected_target, options=expected_options)
 
+    def test_extra_options(self):
+        from six.moves import http_client
+
+        credentials = object()
+        host = 'HOST'
+        user_agent = 'USER_AGENT'
+        extra_options = (('some', 'option'),)
+
+        secure_authorized_channel_patch = mock.patch(
+            'google.auth.transport.grpc.secure_authorized_channel',
+            autospec=True)
+
+        with secure_authorized_channel_patch as secure_authorized_channel:
+            result = self._call_fut(credentials, user_agent, host,
+                                    extra_options)
+
+        self.assertIs(result, secure_authorized_channel.return_value)
+
+        expected_target = '%s:%d' % (host, http_client.HTTPS_PORT)
+        expected_options = (
+            ('grpc.primary_user_agent', user_agent),
+            extra_options[0],
+        )
+
+        secure_authorized_channel.assert_called_once_with(
+            credentials, mock.ANY, expected_target, options=expected_options)
+
 
 class Test_make_secure_stub(unittest.TestCase):
 
@@ -664,13 +691,15 @@ def stub_class(channel):
             channels.append(channel)
             return result
 
-        def mock_channel(*args):
+        def mock_channel(*args, **kwargs):
             channel_args.append(args)
+            channel_args.append(kwargs)
             return channel_obj
 
         credentials = object()
         user_agent = 'you-sir-age-int'
         host = 'localhost'
+        extra_options = {'extra_options': None}
         with _Monkey(MUT, make_secure_channel=mock_channel):
             stub = self._call_fut(credentials, user_agent,
                                   stub_class, host)
@@ -678,7 +707,7 @@ def mock_channel(*args):
         self.assertIs(stub, result)
         self.assertEqual(channels, [channel_obj])
         self.assertEqual(channel_args,
-                         [(credentials, user_agent, host)])
+                         [(credentials, user_agent, host), extra_options])
 
 
 class Test_make_insecure_stub(unittest.TestCase):
diff --git a/system_tests/bigtable.py b/system_tests/bigtable.py
--- a/system_tests/bigtable.py
+++ b/system_tests/bigtable.py
@@ -356,6 +356,23 @@ def _write_to_row(self, row1=None, row2=None, row3=None, row4=None):
         cell4 = Cell(CELL_VAL4, timestamp4)
         return cell1, cell2, cell3, cell4
 
+    def test_read_large_cell_limit(self):
+        row = self._table.row(ROW_KEY)
+        self.rows_to_delete.append(row)
+
+        number_of_bytes = 10 * 1024 * 1024
+        data = b'1' * number_of_bytes  # 10MB of 1's.
+        row.set_cell(COLUMN_FAMILY_ID1, COL_NAME1, data)
+        row.commit()
+
+        # Read back the contents of the row.
+        partial_row_data = self._table.read_row(ROW_KEY)
+        self.assertEqual(partial_row_data.row_key, ROW_KEY)
+        cell = partial_row_data.cells[COLUMN_FAMILY_ID1]
+        column = cell[COL_NAME1]
+        self.assertEqual(len(column), 1)
+        self.assertEqual(column[0].value, data)
+
     def test_read_row(self):
         row = self._table.row(ROW_KEY)
         self.rows_to_delete.append(row)