Repository: googleapis/python-bigquery

Issue:
--use_rest_api still seems to use storage api

#### Environment details
Ubuntu 18.04.5 LTS

Python 3.7.10

pip 21.1.3

google-cloud-bigquery              2.20.0
google-cloud-bigquery-datatransfer 3.2.0
google-cloud-bigquery-storage      2.4.0

#### Steps to reproduce

  1. Use %%bigquery magic with the --use_rest_api flag on a project where I don't bigquery.readsession permissions. 
  2. Observe that the query fails due to not having readsession permissions. 

#### Code example

```python
%%bigquery my_test --use_rest_api
SELECT
*
FROM `aaa-willyn-test.test_dataset.my_test`
```

#### Stack trace
```
PermissionDenied                          Traceback (most recent call last)
<ipython-input-9-f594b2ed9ec4> in <module>
----> 1 get_ipython().run_cell_magic('bigquery', 'my_test --use_rest_api', 'SELECT\n*\nFROM `aaa-willyn-test.test_dataset.abi_copy`\n')

/opt/conda/lib/python3.7/site-packages/IPython/core/interactiveshell.py in run_cell_magic(self, magic_name, line, cell)
   2401             with self.builtin_trap:
   2402                 args = (magic_arg_s, cell)
-> 2403                 result = fn(*args, **kwargs)
   2404             return result
   2405 

...skipping several frames here...

/opt/conda/lib/python3.7/site-packages/google/api_core/grpc_helpers.py in error_remapped_callable(*args, **kwargs)
     67             return callable_(*args, **kwargs)
     68         except grpc.RpcError as exc:
---> 69             six.raise_from(exceptions.from_grpc_error(exc), exc)
     70 
     71     return error_remapped_callable

/opt/conda/lib/python3.7/site-packages/six.py in raise_from(value, from_value)

PermissionDenied: 403 request failed: the user does not have 'bigquery.readsessions.create' permission for 'projects/fc-product-demo'
```




Discussion:
Hmm... at some point the BQ storage API was made a default (before that, BQ Storage API had to be explicitly requested).

I suspect that cell magic just omits that "use BQ Storage", still assuming that REST API will be used by default, but that has changed.

Thanks for reporting this!
Thanks for the context @plamut ! In the meantime, are there any workarounds to get cell magic working with rest API? It doesn't seem to help if I install an older version of google-cloud-bigquery, say 2.14.0
@wnojopra Sorry for missing your question, I went offline for a few days and in the meantime too many notifications have piled up and yours got lost. :/

I can't think of any workarounds from the top of my head other than monekypatching the buggy code, but I'll have a look at this now. Unless it's something really weird and unexpected, you expect the fix some time next week.
@wnojopra Ah, but there IS a workaround - you can pass `--max-results 9999999999` as a cell magic argument. Since BQ Storage is a streaming API, it cannot be used if `max_results` is set and REST API is used instead.

Just make sure that `--max-results` is really big so that it does not silently cut off any result rows.

Resolving patch:
diff --git a/google/cloud/bigquery/magics/magics.py b/google/cloud/bigquery/magics/magics.py
--- a/google/cloud/bigquery/magics/magics.py
+++ b/google/cloud/bigquery/magics/magics.py
@@ -671,7 +671,9 @@ def _cell_magic(line, query):
                 _handle_error(ex, args.destination_var)
                 return
 
-            result = rows.to_dataframe(bqstorage_client=bqstorage_client)
+            result = rows.to_dataframe(
+                bqstorage_client=bqstorage_client, create_bqstorage_client=False,
+            )
             if args.destination_var:
                 IPython.get_ipython().push({args.destination_var: result})
                 return
@@ -728,11 +730,15 @@ def _cell_magic(line, query):
 
         if max_results:
             result = query_job.result(max_results=max_results).to_dataframe(
-                bqstorage_client=bqstorage_client, progress_bar_type=progress_bar
+                bqstorage_client=None,
+                create_bqstorage_client=False,
+                progress_bar_type=progress_bar,
             )
         else:
             result = query_job.to_dataframe(
-                bqstorage_client=bqstorage_client, progress_bar_type=progress_bar
+                bqstorage_client=bqstorage_client,
+                create_bqstorage_client=False,
+                progress_bar_type=progress_bar,
             )
 
         if args.destination_var:


Test patch:
diff --git a/tests/unit/test_magics.py b/tests/unit/test_magics.py
--- a/tests/unit/test_magics.py
+++ b/tests/unit/test_magics.py
@@ -660,7 +660,9 @@ def warning_match(warning):
     assert client_info.user_agent == "ipython-" + IPython.__version__
 
     query_job_mock.to_dataframe.assert_called_once_with(
-        bqstorage_client=bqstorage_instance_mock, progress_bar_type="tqdm"
+        bqstorage_client=bqstorage_instance_mock,
+        create_bqstorage_client=mock.ANY,
+        progress_bar_type="tqdm",
     )
 
     assert isinstance(return_value, pandas.DataFrame)
@@ -703,7 +705,9 @@ def test_bigquery_magic_with_rest_client_requested(monkeypatch):
 
         bqstorage_mock.assert_not_called()
         query_job_mock.to_dataframe.assert_called_once_with(
-            bqstorage_client=None, progress_bar_type="tqdm"
+            bqstorage_client=None,
+            create_bqstorage_client=False,
+            progress_bar_type="tqdm",
         )
 
     assert isinstance(return_value, pandas.DataFrame)
@@ -757,7 +761,12 @@ def test_bigquery_magic_w_max_results_valid_calls_queryjob_result():
         client_query_mock.return_value = query_job_mock
         ip.run_cell_magic("bigquery", "--max_results=5", sql)
 
-        query_job_mock.result.assert_called_with(max_results=5)
+    query_job_mock.result.assert_called_with(max_results=5)
+    query_job_mock.result.return_value.to_dataframe.assert_called_once_with(
+        bqstorage_client=None,
+        create_bqstorage_client=False,
+        progress_bar_type=mock.ANY,
+    )
 
 
 @pytest.mark.usefixtures("ipython_interactive")
@@ -929,7 +938,7 @@ def test_bigquery_magic_w_table_id_and_bqstorage_client():
 
         ip.run_cell_magic("bigquery", "--max_results=5", table_id)
         row_iterator_mock.to_dataframe.assert_called_once_with(
-            bqstorage_client=bqstorage_instance_mock
+            bqstorage_client=bqstorage_instance_mock, create_bqstorage_client=mock.ANY,
         )
 
 
@@ -1246,7 +1255,9 @@ def test_bigquery_magic_w_progress_bar_type_w_context_setter(monkeypatch):
 
         bqstorage_mock.assert_not_called()
         query_job_mock.to_dataframe.assert_called_once_with(
-            bqstorage_client=None, progress_bar_type=magics.context.progress_bar_type
+            bqstorage_client=None,
+            create_bqstorage_client=False,
+            progress_bar_type=magics.context.progress_bar_type,
         )
 
     assert isinstance(return_value, pandas.DataFrame)