Repository: googleapis/python-bigquery

Issue:
`google.cloud.bigquery.job.QueryJob.exception` has wrong behavior
Hello,

According to the [documentation](https://googleapis.dev/python/bigquery/latest/generated/google.cloud.bigquery.job.QueryJob.html#google.cloud.bigquery.job.QueryJob.exception) of `google.cloud.bigquery.job.QueryJob.exception`, this method should returns an exception (and not raises) if something went wrong. But according to my tests this is not always the case:

```python
from google.cloud import bigquery

client = bigquery.Client()

# Correct behavior: returns an exception and does not raise.
query = 'WRONG QUERY'
job = client.query(query)
exc = job.exception()

# Wrong behavior: raises an exception instead of returning it.
query = 'ASSERT (SELECT 2 > 3) AS "Error !!!"'
job = client.query(query)
try:
    job.exception()
except:
    print('raised!')
```

Am I misunderstanding the documentation or is this the correct behavior?

I'm using `google-cloud-bigquery==2.6.1` with Python 3.8.6. Let me know if you need more informations.

Thanks for your help,



Discussion:
@tswast @busunkim96  Please suggest.
I think something is missing for the scenario when `query.state` is `RUNNING` and library wants to retry it, it internally call the `client._get_query_results` method which throws an error of `BadRequest`.
 
I am not sure about to handle this in `bigquery` library and set the `google.api_core.future.polling.PollingFuture.set_exception` from the query.py file as error raised from google-cloud-core http client. 
Working on something related, I discovered that the problem does not exist when doing a double assertion:

```python
# Correct behavior: returns an exception and does not raise.
query = 'ASSERT (SELECT 2 > 3) AS "Error !!!"; ASSERT (SELECT 2 > 3) AS "Error again !!!";'
job = client.query(query)
exc = job.exception()
```

Hope it can help...

Thanks,
@tswast Backend consider "INVALID_ARGUMENT" and gave the reason "invalidQuery" when query contains `ASSERT`, but when query contains only one `ASSERT`  like `query = 'ASSERT (SELECT 2 > 3) AS "Error !!!"'` it returns the `400` status code , hence exception raised from http client, but when query contains multiple `ASSERT`  like `'ASSERT (SELECT 2 > 3) AS "Error !!!"; ASSERT (SELECT 2 > 3) AS "Error again !!!";'`,   it returns `200` status code and  the error result with message. 
Regarding https://github.com/googleapis/python-bigquery/issues/451#issuecomment-751736563 I believe this is a difference in backend API behavior of `getQueryResults` between scripting jobs and non-scripting query jobs.

CC @shollyman
Regarding this issue, it appears that this library may be using the `google-api-core` library slightly incorrectly. The base class (where `exception()` is defined) assumes that `_blocking_poll` doesn't raise any exception

https://github.com/googleapis/python-api-core/blob/b51b7f587042fe9340371c1b5c8e9adf8001c43a/google/api_core/future/polling.py#L129-L134

Instead, it is expected that [set_exception()](https://github.com/googleapis/python-api-core/blob/b51b7f587042fe9340371c1b5c8e9adf8001c43a/google/api_core/future/polling.py#L129-L134) is called.

We do this here:

https://github.com/googleapis/python-bigquery/blob/2849e569d0423e6e40bda953b0e9d38157aaf2df/google/cloud/bigquery/job/base.py#L582

But we are missing the case when the `getQueryResults` backend API throws an error.

We probably need a try-catch block in `QueryJob.done()` as it is not expected (or documented) to raise an exception.

https://github.com/googleapis/python-bigquery/blob/2849e569d0423e6e40bda953b0e9d38157aaf2df/google/cloud/bigquery/job/query.py#L1023

If `_get_query_results` does raise an exception, we might have to inspect it a bit to determine if the job is complete (failed) or it was just a transient issue.
@tswast Has this proposal been confirmed? If yes, I can start implementing it (as there's no assignee on the ticket yet).
Yes. Let's make sure that `exception()` (and also `done()`) does not throw.
`result()` should continue to throw (though it might throw because of `set_exception` instead of the current behavior that bubbles up from `done()`)

Resolving patch:
diff --git a/google/cloud/bigquery/job/query.py b/google/cloud/bigquery/job/query.py
--- a/google/cloud/bigquery/job/query.py
+++ b/google/cloud/bigquery/job/query.py
@@ -989,7 +989,8 @@ def done(self, retry=DEFAULT_RETRY, timeout=None, reload=True):
                 unfinished jobs before checking. Default ``True``.
 
         Returns:
-            bool: True if the job is complete, False otherwise.
+            bool: ``True`` if the job is complete or if fetching its status resulted in
+                an error, ``False`` otherwise.
         """
         # Do not refresh if the state is already done, as the job will not
         # change once complete.
@@ -997,17 +998,34 @@ def done(self, retry=DEFAULT_RETRY, timeout=None, reload=True):
         if not reload or is_done:
             return is_done
 
-        self._reload_query_results(retry=retry, timeout=timeout)
-
         # If an explicit timeout is not given, fall back to the transport timeout
         # stored in _blocking_poll() in the process of polling for job completion.
         transport_timeout = timeout if timeout is not None else self._transport_timeout
 
+        try:
+            self._reload_query_results(retry=retry, timeout=transport_timeout)
+        except exceptions.GoogleAPIError as exc:
+            # Reloading also updates error details on self, thus no need for an
+            # explicit self.set_exception() call if reloading succeeds.
+            try:
+                self.reload(retry=retry, timeout=transport_timeout)
+            except exceptions.GoogleAPIError:
+                # Use the query results reload exception, as it generally contains
+                # much more useful error information.
+                self.set_exception(exc)
+                return True
+            else:
+                return self.state == _DONE_STATE
+
         # Only reload the job once we know the query is complete.
         # This will ensure that fields such as the destination table are
         # correctly populated.
         if self._query_results.complete:
-            self.reload(retry=retry, timeout=transport_timeout)
+            try:
+                self.reload(retry=retry, timeout=transport_timeout)
+            except exceptions.GoogleAPIError as exc:
+                self.set_exception(exc)
+                return True
 
         return self.state == _DONE_STATE
 


Test patch:
diff --git a/tests/unit/job/test_base.py b/tests/unit/job/test_base.py
--- a/tests/unit/job/test_base.py
+++ b/tests/unit/job/test_base.py
@@ -967,7 +967,7 @@ def test_result_w_retry_wo_state(self):
         custom_predicate = mock.Mock()
         custom_predicate.return_value = True
         custom_retry = google.api_core.retry.Retry(
-            predicate=custom_predicate, initial=0.001, maximum=0.001, deadline=0.001,
+            predicate=custom_predicate, initial=0.001, maximum=0.001, deadline=0.1,
         )
         self.assertIs(job.result(retry=custom_retry), job)
 
diff --git a/tests/unit/job/test_query.py b/tests/unit/job/test_query.py
--- a/tests/unit/job/test_query.py
+++ b/tests/unit/job/test_query.py
@@ -16,6 +16,7 @@
 import copy
 import http
 import textwrap
+import types
 
 import freezegun
 from google.api_core import exceptions
@@ -308,7 +309,7 @@ def test_cancelled(self):
 
         self.assertTrue(job.cancelled())
 
-    def test_done(self):
+    def test_done_job_complete(self):
         client = _make_client(project=self.PROJECT)
         resource = self._make_resource(ended=True)
         job = self._get_target_class().from_api_repr(resource, client)
@@ -356,6 +357,84 @@ def test_done_w_timeout_and_longer_internal_api_timeout(self):
         call_args = fake_reload.call_args
         self.assertAlmostEqual(call_args.kwargs.get("timeout"), expected_timeout)
 
+    def test_done_w_query_results_error_reload_ok_job_finished(self):
+        client = _make_client(project=self.PROJECT)
+        bad_request_error = exceptions.BadRequest("Error in query")
+        client._get_query_results = mock.Mock(side_effect=bad_request_error)
+
+        resource = self._make_resource(ended=False)
+        job = self._get_target_class().from_api_repr(resource, client)
+        job._exception = None
+
+        def fake_reload(self, *args, **kwargs):
+            self._properties["status"]["state"] = "DONE"
+            self.set_exception(copy.copy(bad_request_error))
+
+        fake_reload_method = types.MethodType(fake_reload, job)
+
+        with mock.patch.object(job, "reload", new=fake_reload_method):
+            is_done = job.done()
+
+        assert is_done
+        assert isinstance(job._exception, exceptions.BadRequest)
+
+    def test_done_w_query_results_error_reload_ok_job_still_running(self):
+        client = _make_client(project=self.PROJECT)
+        retry_error = exceptions.RetryError("Too many retries", cause=TimeoutError)
+        client._get_query_results = mock.Mock(side_effect=retry_error)
+
+        resource = self._make_resource(ended=False)
+        job = self._get_target_class().from_api_repr(resource, client)
+        job._exception = None
+
+        def fake_reload(self, *args, **kwargs):
+            self._properties["status"]["state"] = "RUNNING"
+
+        fake_reload_method = types.MethodType(fake_reload, job)
+
+        with mock.patch.object(job, "reload", new=fake_reload_method):
+            is_done = job.done()
+
+        assert not is_done
+        assert job._exception is None
+
+    def test_done_w_query_results_error_reload_error(self):
+        client = _make_client(project=self.PROJECT)
+        bad_request_error = exceptions.BadRequest("Error in query")
+        client._get_query_results = mock.Mock(side_effect=bad_request_error)
+
+        resource = self._make_resource(ended=False)
+        job = self._get_target_class().from_api_repr(resource, client)
+        reload_error = exceptions.DataLoss("Oops, sorry!")
+        job.reload = mock.Mock(side_effect=reload_error)
+        job._exception = None
+
+        is_done = job.done()
+
+        assert is_done
+        assert job._exception is bad_request_error
+
+    def test_done_w_job_query_results_ok_reload_error(self):
+        client = _make_client(project=self.PROJECT)
+        query_results = google.cloud.bigquery.query._QueryResults(
+            properties={
+                "jobComplete": True,
+                "jobReference": {"projectId": self.PROJECT, "jobId": "12345"},
+            }
+        )
+        client._get_query_results = mock.Mock(return_value=query_results)
+
+        resource = self._make_resource(ended=False)
+        job = self._get_target_class().from_api_repr(resource, client)
+        retry_error = exceptions.RetryError("Too many retries", cause=TimeoutError)
+        job.reload = mock.Mock(side_effect=retry_error)
+        job._exception = None
+
+        is_done = job.done()
+
+        assert is_done
+        assert job._exception is retry_error
+
     def test_query_plan(self):
         from google.cloud._helpers import _RFC3339_MICROS
         from google.cloud.bigquery.job import QueryPlanEntry
@@ -973,7 +1052,7 @@ def test_result_w_retry(self):
             initial=0.001,
             maximum=0.001,
             multiplier=1.0,
-            deadline=0.001,
+            deadline=0.1,
             predicate=custom_predicate,
         )