Repository: googleapis/python-bigquery

Issue:
`query_and_wait` use in `Cursor.execute` breaks `default_query_job_config` in SQLAlchemy BQ Dialect
#### Environment details

  - OS type and version: Ubuntu 20.04.5 LTS
  - Python version: `python --version` 3.8
  - pip version: `pip --version` 20.0.2
  - `google-cloud-bigquery` version: `pip show google-cloud-bigquery` 3.19.0

#### Steps to reproduce

  1. Run the code in the example
  2. Check job from `query_and_wait` -- it is interactive
  3. Check job from `query` -- it is batch

#### Code example

```python
import google.cloud.bigquery

job_config = google.cloud.bigquery.QueryJobConfig(
    priority=google.cloud.bigquery.QueryPriority.BATCH,
    labels={"test": "test2"}
)
bq_client = google.cloud.bigquery.Client()
bq_client.query_and_wait("SELECT \"query_and_wait\"", job_config=job_config)
bq_client.query("SELECT \"query\"", job_config=job_config)
```

#### Details

`query_and_wait` does not respect all of the `QueryJobConfig` options. This is because it seems to go through a different API (`jobs.query` rather than `jobs.insert`), and `jobs.query` doesn't even seem to respect some of these.

The `query_and_wait` call above ultimately performs a `jobs.query` call with this payload: `{"labels": {"test": "test2"}, "priority": "BATCH", "useLegacySql": false, "formatOptions": {"useInt64Timestamp": true}, "query": "SELECT \\"query_and_wait\\"", "requestId": "..."}`

The `query` call performs this: `'{"jobReference": {"jobId": "...", "projectId": "..."}, "configuration": {"query": {"priority": "BATCH", "useLegacySql": false, "query": "SELECT \\"query\\""}, "labels": {"test": "test2"}}}'`

Note: `priority` is not in the `jobs.query` API, but impact goes beyond this one particular option in `QueryJobConfig`. Labels happens to work because labels happens to be part of the `jobs.query` API.

#### Impact

Although a caller could just migrate from `query_and_wait` to `query` if they were making the call, this is not an option with SQLAlchemy.

The `Cursor.execute` function, from the dbapi, as an optimization, will call `query_and_wait` without a job ID. This *breaks* the ability in SQLAlchemy interactions with BigQuery to supply default query job configs, which is _explicitly_ a goal of the library. ["For situations like these, or for situations where you want the Client to have a default_query_job_config you can pass many arguments in the query of the connection string."](https://github.com/googleapis/python-bigquery-sqlalchemy#:~:text=.%20For%20situations-,like,-these%2C%20or%20for)


Discussion:
@tswast 

Can you take a look at this?
I find it strange that the backend eats unsupported properties in the [request body](https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs/query#QueryRequest), but that does appear to be the behavior.

We do have a denylist of properties we know aren't supported by `jobs.query` here:

https://github.com/googleapis/python-bigquery/blob/b0e95a05d7dbe94f246600ed6c5ebbc2b6a7013f/google/cloud/bigquery/_job_helpers.py#L509-L522

Switching this to an allowlist of properties we know are supported should fix this issue.

Resolving patch:
diff --git a/google/cloud/bigquery/_job_helpers.py b/google/cloud/bigquery/_job_helpers.py
--- a/google/cloud/bigquery/_job_helpers.py
+++ b/google/cloud/bigquery/_job_helpers.py
@@ -400,9 +400,13 @@ def query_and_wait(
             :class:`~google.cloud.bigquery.job.QueryJobConfig`
             class.
     """
+    request_body = _to_query_request(
+        query=query, job_config=job_config, location=location, timeout=api_timeout
+    )
+
     # Some API parameters aren't supported by the jobs.query API. In these
     # cases, fallback to a jobs.insert call.
-    if not _supported_by_jobs_query(job_config):
+    if not _supported_by_jobs_query(request_body):
         return _wait_or_cancel(
             query_jobs_insert(
                 client=client,
@@ -424,9 +428,6 @@ def query_and_wait(
         )
 
     path = _to_query_path(project)
-    request_body = _to_query_request(
-        query=query, job_config=job_config, location=location, timeout=api_timeout
-    )
 
     if page_size is not None and max_results is not None:
         request_body["maxResults"] = min(page_size, max_results)
@@ -506,20 +507,38 @@ def do_query():
         return do_query()
 
 
-def _supported_by_jobs_query(job_config: Optional[job.QueryJobConfig]) -> bool:
+def _supported_by_jobs_query(request_body: Dict[str, Any]) -> bool:
     """True if jobs.query can be used. False if jobs.insert is needed."""
-    if job_config is None:
-        return True
-
-    return (
-        # These features aren't supported by jobs.query.
-        job_config.clustering_fields is None
-        and job_config.destination is None
-        and job_config.destination_encryption_configuration is None
-        and job_config.range_partitioning is None
-        and job_config.table_definitions is None
-        and job_config.time_partitioning is None
-    )
+    request_keys = frozenset(request_body.keys())
+
+    # Per issue: https://github.com/googleapis/python-bigquery/issues/1867
+    # use an allowlist here instead of a denylist because the backend API allows
+    # unsupported parameters without any warning or failure. Instead, keep this
+    # set in sync with those in QueryRequest:
+    # https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs/query#QueryRequest
+    keys_allowlist = {
+        "kind",
+        "query",
+        "maxResults",
+        "defaultDataset",
+        "timeoutMs",
+        "dryRun",
+        "preserveNulls",
+        "useQueryCache",
+        "useLegacySql",
+        "parameterMode",
+        "queryParameters",
+        "location",
+        "formatOptions",
+        "connectionProperties",
+        "labels",
+        "maximumBytesBilled",
+        "requestId",
+        "createSession",
+    }
+
+    unsupported_keys = request_keys - keys_allowlist
+    return len(unsupported_keys) == 0
 
 
 def _wait_or_cancel(


Test patch:
diff --git a/tests/unit/test__job_helpers.py b/tests/unit/test__job_helpers.py
--- a/tests/unit/test__job_helpers.py
+++ b/tests/unit/test__job_helpers.py
@@ -22,6 +22,7 @@
 import pytest
 
 from google.cloud.bigquery.client import Client
+from google.cloud.bigquery import enums
 from google.cloud.bigquery import _job_helpers
 from google.cloud.bigquery.job import copy_ as job_copy
 from google.cloud.bigquery.job import extract as job_extract
@@ -1141,12 +1142,22 @@ def test_make_job_id_w_job_id_overrides_prefix():
             False,
             id="destination_encryption_configuration",
         ),
+        # priority="BATCH" is not supported. See:
+        # https://github.com/googleapis/python-bigquery/issues/1867
+        pytest.param(
+            job_query.QueryJobConfig(
+                priority=enums.QueryPriority.BATCH,
+            ),
+            False,
+            id="priority=BATCH",
+        ),
     ),
 )
-def test_supported_by_jobs_query(
+def test_supported_by_jobs_query_from_queryjobconfig(
     job_config: Optional[job_query.QueryJobConfig], expected: bool
 ):
-    assert _job_helpers._supported_by_jobs_query(job_config) == expected
+    request_body = _job_helpers._to_query_request(job_config, query="SELECT 1")
+    assert _job_helpers._supported_by_jobs_query(request_body) == expected
 
 
 def test_wait_or_cancel_no_exception():