Repository: googleapis/python-bigquery

Issue:
Returned API errors are an HTML rendering, causing exception in client
If an error occurs whilst inserting rows, an exception is raised by the client with rendered HTML, instead of the errors being returned as documented [here](https://googleapis.dev/python/bigquery/latest/generated/google.cloud.bigquery.client.Client.html#google.cloud.bigquery.client.Client.insert_rows).

This is probably an integration testing problem. I guess that the BigQuery API has changed its default error response format to rendered HTML since this code was written. It may alternatively be that the client only handles row-level errors correctly but raises non-row-level exceptions. If the latter, then:
- these additional raises should be documented (current docs say `:raises: ValueError – if table’s schema is not set or rows is not a Sequence`), and
- the error raised should include meaningful information, not just`<!DOCTYPE html>`

#### Environment details

  - OS type and version: whatever google cloud functions run on (debian?)
  - Python version: 3.9
  - pip version: n/a
  - `google-cloud-bigquery` version: 2.34.3

#### Steps to reproduce

**NOTE: I'm only assuming this is a problem with maximum row size, due to the 413 error I receive and the fact that the same code succeeds on smaller inserts. It has to be an assumption because the whole problem is I can't see the error!**

  1.  Prepare a table (in this case I call it sensor_data) with whatever schema
  2. Using the following code sample... https://cloud.google.com/bigquery/docs/samples/bigquery-table-insert-rows
  3. Insert data into the table using `client.insert_rows`:
```py
rows = [
    # ...create a lot of data, enough to violate the limits here: https://cloud.google.com/bigquery/quotas#streaming_inserts
]
client = bigquery.Client()
table=self.client.get_table(self.table_names["sensor_data"])
errors = client.insert_rows(table=table, rows=rows)
```

#### Stack trace
```
Traceback (most recent call last): 
File "/layers/google.python.pip/pip/lib/python3.9/site-packages/flask/app.py", line 2073, in wsgi_app response = self.full_dispatch_request()
File "/layers/google.python.pip/pip/lib/python3.9/site-packages/flask/app.py", line 1518, in full_dispatch_request rv = self.handle_user_exception(e) 
File "/layers/google.python.pip/pip/lib/python3.9/site-packages/flask/app.py", line 1516, in full_dispatch_request rv = self.dispatch_request() 
File "/layers/google.python.pip/pip/lib/python3.9/site-packages/flask/app.py", line 1502, in dispatch_request return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) 
File "/layers/google.python.pip/pip/lib/python3.9/site-packages/functions_framework/__init__.py", line 171, in view_func function(data, context) 
File "/workspace/main.py", line 38, in upload_window window_handler.persist_window(window, window_metadata) 
File "/workspace/window_handler.py", line 91, in persist_window self.dataset.add_sensor_data( 
File "/workspace/big_query.py", line 99, in add_sensor_data errors = self.client.insert_rows(table=self.client.get_table(self.table_names["sensor_data"]), rows=rows) 
File "/layers/google.python.pip/pip/lib/python3.9/site-packages/google/cloud/bigquery/client.py", line 3466, in insert_rows return self.insert_rows_json(table, json_rows, **kwargs) 
File "/layers/google.python.pip/pip/lib/python3.9/site-packages/google/cloud/bigquery/client.py", line 3646, in insert_rows_json response = self._call_api( 
File "/layers/google.python.pip/pip/lib/python3.9/site-packages/google/cloud/bigquery/client.py", line 782, in _call_api return call() 
File "/layers/google.python.pip/pip/lib/python3.9/site-packages/google/api_core/retry.py", line 283, in retry_wrapped_func return retry_target( 
File "/layers/google.python.pip/pip/lib/python3.9/site-packages/google/api_core/retry.py", line 190, in retry_target return target() 
File "/layers/google.python.pip/pip/lib/python3.9/site-packages/google/cloud/_http/__init__.py", line 494, in api_request raise exceptions.from_http_response(response) google.api_core.exceptions.GoogleAPICall

Error: 413 POST https://bigquery.googleapis.com/bigquery/v2/projects/aerosense-twined/datasets/greta/tables/sensor_data/insertAll?prettyPrint=false: <!DOCTYPE html>
```

This prevents us from ever seeing what the actual error is that would be shown on the page.

Thanks!



Discussion:
This is best addressed by augmenting the docs for the python library.

Additional context:  If you try to insert vastly oversized volumes of data in a single request, the error (and the html output) is emitted by intermediate components in the serving path.

There's an existing test in the go library that tests this behavior: https://github.com/googleapis/google-cloud-go/blob/5b7bfebcd4a0fd3cbe355d9d290e6b5101810b7e/bigquery/integration_test.go#L841

Assigning to one of the python maintainers to follow up here.


Resolving patch:
diff --git a/google/cloud/bigquery/client.py b/google/cloud/bigquery/client.py
--- a/google/cloud/bigquery/client.py
+++ b/google/cloud/bigquery/client.py
@@ -3358,6 +3358,14 @@ def insert_rows(
         See
         https://cloud.google.com/bigquery/docs/reference/rest/v2/tabledata/insertAll
 
+        BigQuery will reject insertAll payloads that exceed a defined limit (10MB).
+        Additionally, if a payload vastly exceeds this limit, the request is rejected
+        by the intermediate architecture, which returns a 413 (Payload Too Large) status code.
+
+
+        See
+        https://cloud.google.com/bigquery/quotas#streaming_inserts
+
         Args:
             table (Union[ \
                 google.cloud.bigquery.table.Table, \
@@ -3424,6 +3432,13 @@ def insert_rows_from_dataframe(
     ) -> Sequence[Sequence[dict]]:
         """Insert rows into a table from a dataframe via the streaming API.
 
+        BigQuery will reject insertAll payloads that exceed a defined limit (10MB).
+        Additionally, if a payload vastly exceeds this limit, the request is rejected
+        by the intermediate architecture, which returns a 413 (Payload Too Large) status code.
+
+        See
+        https://cloud.google.com/bigquery/quotas#streaming_inserts
+
         Args:
             table (Union[ \
                 google.cloud.bigquery.table.Table, \
@@ -3485,6 +3500,13 @@ def insert_rows_json(
         See
         https://cloud.google.com/bigquery/docs/reference/rest/v2/tabledata/insertAll
 
+        BigQuery will reject insertAll payloads that exceed a defined limit (10MB).
+        Additionally, if a payload vastly exceeds this limit, the request is rejected
+        by the intermediate architecture, which returns a 413 (Payload Too Large) status code.
+
+        See
+        https://cloud.google.com/bigquery/quotas#streaming_inserts
+
         Args:
             table (Union[ \
                 google.cloud.bigquery.table.Table \


Test patch: