Repository: googleapis/google-cloud-python

Issue:
BigQuery: Let `bqstorage_client` authentication errors be raised in `to_dataframe`
Now that I'm trying to write a tutorial on using the BigQuery Storage API to download query results, I've had a change of heart regarding whether `to_dataframe` should fall back to the `tabledata.list` API when there are authentication errors. Originally, I thought we should fallback on authentication errors, but now I see that it hides problems.

Someone could have downloaded the right libraries but either don't have the right permissions or haven't enabled the API, but those errors will be swallowed. Instead, downloading large tables will be much slower than expected.


Discussion:
I think this is worthy of being a bug. I consider it a "known issue" for `to_dataframe` that authorization errors can be swallowed.

Resolving patch:
diff --git a/bigquery/google/cloud/bigquery/table.py b/bigquery/google/cloud/bigquery/table.py
--- a/bigquery/google/cloud/bigquery/table.py
+++ b/bigquery/google/cloud/bigquery/table.py
@@ -1468,6 +1468,12 @@ def to_dataframe(self, bqstorage_client=None, dtypes=None, progress_bar_type=Non
         if bqstorage_client is not None:
             try:
                 return self._to_dataframe_bqstorage(bqstorage_client, dtypes)
+            except google.api_core.exceptions.Forbidden:
+                # Don't hide errors such as insufficient permissions to create
+                # a read session, or the API is not enabled. Both of those are
+                # clearly problems if the developer has explicitly asked for
+                # BigQuery Storage API support.
+                raise
             except google.api_core.exceptions.GoogleAPICallError:
                 # There is a known issue with reading from small anonymous
                 # query results tables, so some errors are expected. Rather


Test patch:
diff --git a/bigquery/tests/unit/test_table.py b/bigquery/tests/unit/test_table.py
--- a/bigquery/tests/unit/test_table.py
+++ b/bigquery/tests/unit/test_table.py
@@ -1793,6 +1793,28 @@ def test_to_dataframe_w_bqstorage_fallback_to_tabledata_list(self):
         self.assertEqual(df.name.dtype.name, "object")
         self.assertEqual(df.age.dtype.name, "int64")
 
+    @unittest.skipIf(pandas is None, "Requires `pandas`")
+    @unittest.skipIf(
+        bigquery_storage_v1beta1 is None, "Requires `google-cloud-bigquery-storage`"
+    )
+    def test_to_dataframe_w_bqstorage_raises_auth_error(self):
+        from google.cloud.bigquery import table as mut
+
+        bqstorage_client = mock.create_autospec(
+            bigquery_storage_v1beta1.BigQueryStorageClient
+        )
+        bqstorage_client.create_read_session.side_effect = google.api_core.exceptions.Forbidden(
+            "TEST BigQuery Storage API not enabled. TEST"
+        )
+        path = "/foo"
+        api_request = mock.Mock(return_value={"rows": []})
+        row_iterator = mut.RowIterator(
+            _mock_client(), api_request, path, [], table=mut.Table("proj.dset.tbl")
+        )
+
+        with pytest.raises(google.api_core.exceptions.Forbidden):
+            row_iterator.to_dataframe(bqstorage_client=bqstorage_client)
+
     @unittest.skipIf(
         bigquery_storage_v1beta1 is None, "Requires `google-cloud-bigquery-storage`"
     )