Repository: googleapis/google-cloud-python

Issue:
Setting a timeout for BigQuery async_query doesn't work
The documentation and code appears to allow for setting a maximum duration before timing out for an async_query by passing `timeout` (an int for the number of milliseconds) to the `result()` function that is called on a query job (https://github.com/GoogleCloudPlatform/google-cloud-python/blob/master/bigquery/google/cloud/bigquery/job.py#L476 and https://github.com/GoogleCloudPlatform/google-cloud-python/blob/master/bigquery/google/cloud/bigquery/job.py#L1311). Similarly, there also appear to be some references to the `fetch_data()` function of a QueryResults object being able to accept a timeout parameter (https://github.com/GoogleCloudPlatform/google-cloud-python/blob/master/bigquery/google/cloud/bigquery/query.py#L390).

However, passing a timeout value of 1 or 0 in either fashion with a async_query does nothing to interrupt the query from being completed. Queries complete regardless and output results:

```
from google.cloud import bigquery
import uuid
client = bigquery.Client(project=project)
query_job = client.run_async_query(str(uuid.uuid4()),
                                   "select 'hello' as a, 32423432 as b")
query_job.begin()
query_results = query_job.result(timeout=1)
data = query_results.fetch_data(timeout_ms=1)
rows = list(data)
print(rows)
```
Output:
```
[(u'hello', 32423432)]
```

By contrast, setting a timeout value for a sync_query by setting the `timeout_ms` property of a QueryJob indeed works as expected as raises the appropriate JobComplete = False flag:

```
from google.cloud import bigquery
client = bigquery.Client(project=project)
query_job = client.run_sync_query("select 'hello' as a, 32423432 as b")
query_job.timeout_ms=1
query_job.run()
print(query_job._properties.get("jobComplete"))
rows = list(query_job.rows)
print(rows)
```
Output:
```
False
[]
```

It appears timeout for async_queries is still to be implemented and it is unclear how to access a JobIncomplete property via an async_query, forcing users to continue using sync_queries if they want the ability to time out their queries.

cc: @tswast who requested I raise the issue which I identified in this pandas-gbq PR: https://github.com/pydata/pandas-gbq/pull/25#discussion_r143263462

OSX
Python 2.7.13
google-cloud-python 0.27.0 and google-cloud-bigquery 0.26.0



Discussion:
/cc @jonparrott 
Just to clarify, the `timeout` parameter on your last `run_sync_query` example does not set a timeout for the query. It sets a timeout on waiting for the results. If the request times out, the query is still running in the background. You can verify this by looking at the list of jobs created.

You have to run `QueryJob.cancel()` to actually stop the query, and even that is only best-effort because the query could theoretically complete before the cancel request propagates to the query workers.

One reason for the behavior you are experiencing is that the `result` timeout is in seconds and the provided query will almost certainly complete within 1 second. See: https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.Future.result
I'm trying to provide back compatibility for this feature: https://github.com/pydata/pandas-gbq/pull/76. This is for preventing stalled jobs from hanging an entire script, and to have an elegant way to move on in the script without having to write our own timer function to handle; if the actual query can be killed in background, all the better, but at the bare minimum, it should stop looking for results after a set amount of time and move on. Unfortunately I can't replicate with aync_query, but I can with sync_query. 

As for seconds, I tried several more complex queries that definitely takes more than 1 second to complete and it still returned results for all of them. I'm ok with updating to seconds in my project since I can't imagine many use cases for sub-1 second timeouts being needed, but if it is indeed supposed to work this way, the documentation here should update to reflect this since there are no mentions of the timeout parameter being in seconds (and for legacy users and those perusing the docs, one might be mistaken that it's supposed to be milliseconds due to the timeoutMs and timeout_ms references).
I think I know what's going on. The call to result repeatedly calls the done method, but since the done method does an API that could last up to 10 seconds, any timeout less than 10 seconds won't work.

@jonparrot Do you think we could add an optional timeout parameter to done? Or a should we pass the timeout from result to done in some other way?
@tswast I don't think we should use connection/response timeouts to indicate incomplete. If we are going to poll indefinitely for the response then blocking on the request is fine, but if the poll has a deadline then the request to check if a job is done should return as quickly as possible.
Since the deadline is an integer in seconds, maybe it would make sense if `done()` always tries to complete within 1 second (as compared to the current default of 10 seconds)?

Resolving patch:
diff --git a/bigquery/google/cloud/bigquery/client.py b/bigquery/google/cloud/bigquery/client.py
--- a/bigquery/google/cloud/bigquery/client.py
+++ b/bigquery/google/cloud/bigquery/client.py
@@ -1108,10 +1108,10 @@ def query_rows(self, query, job_config=None, job_id=None, timeout=None,
         :type job_id: str
         :param job_id: (Optional) ID to use for the query job.
 
-        :type timeout: int
+        :type timeout: float
         :param timeout:
-            (Optional) How long to wait for job to complete before raising a
-            :class:`TimeoutError`.
+            (Optional) How long (in seconds) to wait for job to complete
+            before raising a :class:`TimeoutError`.
 
         :rtype: :class:`~google.api.core.page_iterator.Iterator`
         :returns:
diff --git a/bigquery/google/cloud/bigquery/job.py b/bigquery/google/cloud/bigquery/job.py
--- a/bigquery/google/cloud/bigquery/job.py
+++ b/bigquery/google/cloud/bigquery/job.py
@@ -44,6 +44,7 @@
 
 _DONE_STATE = 'DONE'
 _STOPPED_REASON = 'stopped'
+_TIMEOUT_BUFFER_SECS = 0.1
 
 _ERROR_REASON_TO_EXCEPTION = {
     'accessDenied': http_client.FORBIDDEN,
@@ -1522,6 +1523,7 @@ def __init__(self, job_id, query, client, job_config=None):
         self.query = query
         self._configuration = job_config
         self._query_results = None
+        self._done_timeout = None
 
     @property
     def allow_large_results(self):
@@ -1871,7 +1873,7 @@ def query_results(self, retry=DEFAULT_RETRY):
         """
         if not self._query_results:
             self._query_results = self._client._get_query_results(
-                self.job_id, retry)
+                self.job_id, retry, project=self.project)
         return self._query_results
 
     def done(self, retry=DEFAULT_RETRY):
@@ -1880,11 +1882,25 @@ def done(self, retry=DEFAULT_RETRY):
         :rtype: bool
         :returns: True if the job is complete, False otherwise.
         """
+        # Since the API to getQueryResults can hang up to the timeout value
+        # (default of 10 seconds), set the timeout parameter to ensure that
+        # the timeout from the futures API is respected. See:
+        # https://github.com/GoogleCloudPlatform/google-cloud-python/issues/4135
+        timeout_ms = None
+        if self._done_timeout is not None:
+            # Subtract a buffer for context switching, network latency, etc.
+            timeout = self._done_timeout - _TIMEOUT_BUFFER_SECS
+            timeout = max(min(timeout, 10), 0)
+            self._done_timeout -= timeout
+            self._done_timeout = max(0, self._done_timeout)
+            timeout_ms = int(timeout * 1000)
+
         # Do not refresh is the state is already done, as the job will not
         # change once complete.
         if self.state != _DONE_STATE:
             self._query_results = self._client._get_query_results(
-                self.job_id, retry)
+                self.job_id, retry,
+                project=self.project, timeout_ms=timeout_ms)
 
             # Only reload the job once we know the query is complete.
             # This will ensure that fields such as the destination table are
@@ -1894,10 +1910,14 @@ def done(self, retry=DEFAULT_RETRY):
 
         return self.state == _DONE_STATE
 
+    def _blocking_poll(self, timeout=None):
+        self._done_timeout = timeout
+        super(QueryJob, self)._blocking_poll(timeout=timeout)
+
     def result(self, timeout=None, retry=DEFAULT_RETRY):
         """Start the job and wait for it to complete and get the result.
 
-        :type timeout: int
+        :type timeout: float
         :param timeout:
             How long to wait for job to complete before raising a
             :class:`TimeoutError`.


Test patch:
diff --git a/bigquery/tests/unit/test_job.py b/bigquery/tests/unit/test_job.py
--- a/bigquery/tests/unit/test_job.py
+++ b/bigquery/tests/unit/test_job.py
@@ -2197,6 +2197,34 @@ def test_result_invokes_begins(self):
         self.assertEqual(query_request['method'], 'GET')
         self.assertEqual(reload_request['method'], 'GET')
 
+    def test_result_w_timeout(self):
+        begun_resource = self._makeResource()
+        query_resource = {
+            'jobComplete': True,
+            'jobReference': {
+                'projectId': self.PROJECT,
+                'jobId': self.JOB_ID,
+            },
+        }
+        done_resource = copy.deepcopy(begun_resource)
+        done_resource['status'] = {'state': 'DONE'}
+        connection = _Connection(
+            begun_resource, query_resource, done_resource)
+        client = _make_client(project=self.PROJECT, connection=connection)
+        job = self._make_one(self.JOB_ID, self.QUERY, client)
+
+        job.result(timeout=1.0)
+
+        self.assertEqual(len(connection._requested), 3)
+        begin_request, query_request, reload_request = connection._requested
+        self.assertEqual(begin_request['method'], 'POST')
+        self.assertEqual(query_request['method'], 'GET')
+        self.assertEqual(
+            query_request['path'],
+            '/projects/{}/queries/{}'.format(self.PROJECT, self.JOB_ID))
+        self.assertEqual(query_request['query_params']['timeoutMs'], 900)
+        self.assertEqual(reload_request['method'], 'GET')
+
     def test_result_error(self):
         from google.cloud import exceptions