Repository: googleapis/python-bigquery

Issue:
pd.read_gbq broken with 1.26.0 and pyarrow
If pyarrow is installed, then with pandas-gbq==0.13.2, using `pd.read_gbq` causes an exception inside this library.

```
>>> pd.read_gbq("SELECT 1", project_id="ebmdatalab")
/home/inglesp/.pyenv/versions/openp/lib/python3.5/site-packages/google/auth/_default.py:69: UserWarning: Your application has authenticated using end user credentials from Google Cloud SDK without a quota project. You might receive a "quota exceeded" or "API not enabled" error. We recommend you rerun `gcloud auth application-default login` and make sure a quota project is added. Or you can use service accounts instead. For more information about service accounts, see https://cloud.google.com/docs/authentication/
  warnings.warn(_CLOUD_SDK_CREDENTIALS_WARNING)
/home/inglesp/.pyenv/versions/openp/lib/python3.5/site-packages/google/cloud/bigquery/client.py:407: UserWarning: Cannot create BigQuery Storage client, the dependency google-cloud-bigquery-storage is not installed.
  "Cannot create BigQuery Storage client, the dependency "
Downloading: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.23rows/s]
Traceback (most recent call last):
  File "/home/inglesp/.pyenv/versions/3.5.9/lib/python3.5/code.py", line 91, in runcode
    exec(code, self.locals)
  File "<console>", line 1, in <module>
  File "/home/inglesp/.pyenv/versions/openp/lib/python3.5/site-packages/pandas/io/gbq.py", line 176, in read_gbq
    **kwargs
  File "/home/inglesp/.pyenv/versions/openp/lib/python3.5/site-packages/pandas_gbq/gbq.py", line 967, in read_gbq
    progress_bar_type=progress_bar_type,
  File "/home/inglesp/.pyenv/versions/openp/lib/python3.5/site-packages/pandas_gbq/gbq.py", line 532, in run_query
    progress_bar_type=progress_bar_type,
  File "/home/inglesp/.pyenv/versions/openp/lib/python3.5/site-packages/pandas_gbq/gbq.py", line 562, in _download_results
    progress_bar_type=progress_bar_type,
  File "/home/inglesp/.pyenv/versions/openp/lib/python3.5/site-packages/google/cloud/bigquery/table.py", line 1727, in to_dataframe
    create_bqstorage_client=create_bqstorage_client,
  File "/home/inglesp/.pyenv/versions/openp/lib/python3.5/site-packages/google/cloud/bigquery/table.py", line 1561, in to_arrow
    bqstorage_client.transport.channel.close()
AttributeError: 'NoneType' object has no attribute 'transport'
```


If pyarrow is not installed, there is no exception.  The same code works with 1.25.0, so I'm raising the issue against this library and not [pydata/pandas-gbq/](https://github.com/pydata/pandas-gbq/) or [apache/arrow](https://github.com/apache/arrow).

Here are details of the various versions used to reproduce this.

```
$ python --version
Python 3.8.2
$ cat requirements.in 
google-cloud-bigquery
pandas-gbq
pyarrow
$ pip freeze
cachetools==4.1.1
certifi==2020.6.20
chardet==3.0.4
click==7.1.2
google-api-core==1.22.0
google-auth==1.19.2
google-auth-oauthlib==0.4.1
google-cloud-bigquery==1.26.0
google-cloud-core==1.3.0
google-resumable-media==0.5.1
googleapis-common-protos==1.52.0
idna==2.10
numpy==1.19.1
oauthlib==3.1.0
pandas==1.0.5
pandas-gbq==0.13.2
pip-tools==5.2.1
protobuf==3.12.2
pyarrow==0.17.1
pyasn1==0.4.8
pyasn1-modules==0.2.8
pydata-google-auth==1.1.0
python-dateutil==2.8.1
pytz==2020.1
requests==2.24.0
requests-oauthlib==1.3.0
rsa==4.6
six==1.15.0
urllib3==1.25.9
```


Discussion:
I think I see what the issue is. The `to_arrow()` method tries to create the BQ storage client, which fails due to the missing optional dependency. However, the `owns_bqstorage_client` flag is still set to `True` even though the BQ Storage client could not be constructed, and later on the method tries to close the transport on it, which fails, because client is `None`.

If feasible, one possible quick workaround would be to install the `google-cloud-bigquery-storage` dependency (that client is faster anyway).
> If feasible, one possible quick workaround would be to install the google-cloud-bigquery-storage dependency (that client is faster anyway).

Are there any downsides to using google-cloud-bigquery-storage?
@inglesp On a technical level probably not, apart from a few extra dependencies. It's also _much_ faster, especially for large datasets. On the downside, however,  the [BQ Storage API](https://cloud.google.com/bigquery/docs/reference/storage/) is billable (check the link in the Pricing section at the end), which can affect any business aspects of a project.

@shollyman Are there perhaps any other factors that should be taken into account?


The [release notes of 1.26.0](https://github.com/googleapis/python-bigquery/releases) say:

> use BigQuery Storage client by default

Shouldn't it be added to the dependencies in setup.py then? 
@bartaelterman It is added, but it's an optional dependency (the BQ Storage client has reached the stable version only recently). The release notes should probably have emphasized "if BQ Storage client is available", though.
> @shollyman Are there perhaps any other factors that should be taken into account?

Differences in billing are really the major consideration if this integration is the major use case.  The storage API has additional features (projection/filtering/snapshot control), but they're for use cases where you desrie custom processing of the manged storage and you don't want the BigQuery query engine to do any of the work.

API enablement is mirrored/shared, so no differences there.


Resolving patch:
diff --git a/google/cloud/bigquery/table.py b/google/cloud/bigquery/table.py
--- a/google/cloud/bigquery/table.py
+++ b/google/cloud/bigquery/table.py
@@ -1534,8 +1534,8 @@ def to_arrow(
 
         owns_bqstorage_client = False
         if not bqstorage_client and create_bqstorage_client:
-            owns_bqstorage_client = True
             bqstorage_client = self.client._create_bqstorage_client()
+            owns_bqstorage_client = bqstorage_client is not None
 
         try:
             progress_bar = self._get_progress_bar(progress_bar_type)


Test patch:
diff --git a/tests/unit/test_table.py b/tests/unit/test_table.py
--- a/tests/unit/test_table.py
+++ b/tests/unit/test_table.py
@@ -1990,6 +1990,33 @@ def test_to_arrow_w_bqstorage_creates_client(self):
         mock_client._create_bqstorage_client.assert_called_once()
         bqstorage_client.transport.channel.close.assert_called_once()
 
+    @unittest.skipIf(pyarrow is None, "Requires `pyarrow`")
+    def test_to_arrow_create_bqstorage_client_wo_bqstorage(self):
+        from google.cloud.bigquery.schema import SchemaField
+
+        schema = [
+            SchemaField("name", "STRING", mode="REQUIRED"),
+            SchemaField("age", "INTEGER", mode="REQUIRED"),
+        ]
+        rows = [
+            {"f": [{"v": "Alice"}, {"v": "98"}]},
+            {"f": [{"v": "Bob"}, {"v": "99"}]},
+        ]
+        path = "/foo"
+        api_request = mock.Mock(return_value={"rows": rows})
+
+        mock_client = _mock_client()
+        mock_client._create_bqstorage_client.return_value = None
+        row_iterator = self._make_one(mock_client, api_request, path, schema)
+
+        tbl = row_iterator.to_arrow(create_bqstorage_client=True)
+
+        # The client attempted to create a BQ Storage client, and even though
+        # that was not possible, results were still returned without errors.
+        mock_client._create_bqstorage_client.assert_called_once()
+        self.assertIsInstance(tbl, pyarrow.Table)
+        self.assertEqual(tbl.num_rows, 2)
+
     @unittest.skipIf(pyarrow is None, "Requires `pyarrow`")
     @unittest.skipIf(
         bigquery_storage_v1 is None, "Requires `google-cloud-bigquery-storage`"