Repository: googleapis/python-bigquery

Issue:
Random timeouts in api calls to bigquery.googleapis.com
After updating google-cloud-bigquery from version 1.19.0 to 1.24.0, 
`requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='bigquery.googleapis.com', port=443): Read timed out. (read timeout=11.0)` started to pop up randomly.

This is likely related to https://github.com/googleapis/python-bigquery/issues/34

#### Environment details

  - OS type and version: `Ubuntu 19.10 x64`
  - Python version: `3.7.5`
  - pip version: `pip 20.0.2`
  - `google-cloud-bigquery` version: `1.24.0`

#### Steps to reproduce

The error is not deterministic, however we observed it on production environments (compute instances in GCP) and local developement environments.

For about 100 queries, at least 1-2 fails with this error (which makes it reproducible). This makes something like 1-2% of ALL requests to fail!

If I understand correctly, BQ API endpoint responsible for result() method will block for at most 10 seconds. There is also 1 second margin to neutralize network lags etc. No retry mechanism covering the timeout is present, so in case of delay of more than 1 second, the whole request will fail.

In my opinion, having 1.0s non-configurable timeout is not safe. Also not-mutating endpoints (like "is job finished") should automatically retry in case of timeout. This is not implemented at the moment.

We had to rollback to 1.19.0 to make everything stable again.

#### Code example

Nothing really helpful could be placed here.

The easiest way to reproduce this error is to run query that takes MORE than 10s in 100x loop.

#### Stack trace

The stack trace is always the same:
```
Traceback (most recent call last):
  File "<proj>/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 421, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "<proj>/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 416, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib/python3.7/http/client.py", line 1344, in getresponse
    response.begin()
  File "/usr/lib/python3.7/http/client.py", line 306, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.7/http/client.py", line 267, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/lib/python3.7/socket.py", line 589, in readinto
    return self._sock.recv_into(b)
  File "/usr/lib/python3.7/ssl.py", line 1071, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/lib/python3.7/ssl.py", line 929, in read
    return self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<proj>/venv/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "<proj>/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 720, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "<proj>/venv/lib/python3.7/site-packages/urllib3/util/retry.py", line 400, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "<proj>/venv/lib/python3.7/site-packages/urllib3/packages/six.py", line 735, in reraise
    raise value
  File "<proj>/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 672, in urlopen
    chunked=chunked,
  File "<proj>/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 423, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File "<proj>/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 331, in _raise_timeout
    self, url, "Read timed out. (read timeout=%s)" % timeout_value
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='bigquery.googleapis.com', port=443): Read timed out. (read timeout=11.0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<proj>/venv/lib/python3.7/site-packages/google/cloud/bigquery/job.py", line 3196, in result
    super(QueryJob, self).result(retry=retry, timeout=timeout)
  File "<proj>/venv/lib/python3.7/site-packages/google/cloud/bigquery/job.py", line 818, in result
    return super(_AsyncJob, self).result(timeout=timeout)
  File "<proj>/venv/lib/python3.7/site-packages/google/api_core/future/polling.py", line 122, in result
    self._blocking_poll(timeout=timeout)
  File "<proj>/venv/lib/python3.7/site-packages/google/cloud/bigquery/job.py", line 3098, in _blocking_poll
    super(QueryJob, self)._blocking_poll(timeout=timeout)
  File "<proj>/venv/lib/python3.7/site-packages/google/api_core/future/polling.py", line 101, in _blocking_poll
    retry_(self._done_or_raise)()
  File "<proj>/venv/lib/python3.7/site-packages/google/api_core/retry.py", line 289, in retry_wrapped_func
    return retry_wrapped_func
  File "<proj>/venv/lib/python3.7/site-packages/google/api_core/retry.py", line 184, in retry_target
    return target()
  File "<proj>/venv/lib/python3.7/site-packages/google/api_core/future/polling.py", line 80, in _done_or_raise
    if not self.done():
  File "<proj>/venv/lib/python3.7/site-packages/google/cloud/bigquery/job.py", line 3085, in done
    timeout=timeout,
  File "<proj>/venv/lib/python3.7/site-packages/google/cloud/bigquery/client.py", line 1287, in _get_query_results
    retry, method="GET", path=path, query_params=extra_params, timeout=timeout
  File "<proj>/venv/lib/python3.7/site-packages/google/cloud/bigquery/client.py", line 556, in _call_api
    return call()
  File "<proj>/venv/lib/python3.7/site-packages/google/api_core/retry.py", line 289, in retry_wrapped_func
    return retry_wrapped_func
  File "<proj>/venv/lib/python3.7/site-packages/google/api_core/retry.py", line 184, in retry_target
    return target()
  File "<proj>/venv/lib/python3.7/site-packages/google/cloud/_http.py", line 419, in api_request
    timeout=timeout,
  File "<proj>/venv/lib/python3.7/site-packages/google/cloud/_http.py", line 277, in _make_request
    method, url, headers, data, target_object, timeout=timeout
  File "<proj>/venv/lib/python3.7/site-packages/google/cloud/_http.py", line 315, in _do_request
    url=url, method=method, headers=headers, data=data, timeout=timeout
  File "<proj>/venv/lib/python3.7/site-packages/google/auth/transport/requests.py", line 317, in request
    **kwargs
  File "<proj>/venv/lib/python3.7/site-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "<proj>/venv/lib/python3.7/site-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "<proj>/venv/lib/python3.7/site-packages/requests/adapters.py", line 529, in send
    raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='bigquery.googleapis.com', port=443): Read timed out. (read timeout=11.0)
```


Discussion:
In the past versions there was a problem with requests sometimes stalling indefinitely, because by default no timeouts were used for the transport layer. After introducing the default timeouts, these sometimes kick in too aggressively, because a transport timeout is not independent from the [timeoutMs](https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs/getQueryResults#query-parameters) that is used when polling the server whether a query job has completed.

The latter timeout has a maximum of 10 seconds, and if the job is not done by, an internal `_OperationNotComplete()` error is raised, which is automatically retried until the job is done (or the termination condition is fulfilled).

Now, using that same timeout as the timeout for the request itself (at the transport layer) does not always work that well, and these two should probably be separated. However, there still needs be some transport timeout (in order to not block indefinitely).

I'll probably have time too take a closer look at this next week.

Resolving patch:
diff --git a/google/cloud/bigquery/client.py b/google/cloud/bigquery/client.py
--- a/google/cloud/bigquery/client.py
+++ b/google/cloud/bigquery/client.py
@@ -22,7 +22,6 @@
 except ImportError:  # Python 2.7
     import collections as collections_abc
 
-import concurrent.futures
 import copy
 import functools
 import gzip
@@ -48,7 +47,6 @@
 import google.api_core.client_options
 import google.api_core.exceptions
 from google.api_core import page_iterator
-from google.auth.transport.requests import TimeoutGuard
 import google.cloud._helpers
 from google.cloud import exceptions
 from google.cloud.client import ClientWithProject
@@ -2598,27 +2596,22 @@ def list_partitions(self, table, retry=DEFAULT_RETRY, timeout=None):
             timeout (Optional[float]):
                 The number of seconds to wait for the underlying HTTP transport
                 before using ``retry``.
-                If multiple requests are made under the hood, ``timeout`` is
-                interpreted as the approximate total time of **all** requests.
+                If multiple requests are made under the hood, ``timeout``
+                applies to each individual request.
 
         Returns:
             List[str]:
                 A list of the partition ids present in the partitioned table
         """
         table = _table_arg_to_table_ref(table, default_project=self.project)
-
-        with TimeoutGuard(
-            timeout, timeout_error_type=concurrent.futures.TimeoutError
-        ) as guard:
-            meta_table = self.get_table(
-                TableReference(
-                    DatasetReference(table.project, table.dataset_id),
-                    "%s$__PARTITIONS_SUMMARY__" % table.table_id,
-                ),
-                retry=retry,
-                timeout=timeout,
-            )
-        timeout = guard.remaining_timeout
+        meta_table = self.get_table(
+            TableReference(
+                DatasetReference(table.project, table.dataset_id),
+                "%s$__PARTITIONS_SUMMARY__" % table.table_id,
+            ),
+            retry=retry,
+            timeout=timeout,
+        )
 
         subset = [col for col in meta_table.schema if col.name == "partition_id"]
         return [
@@ -2685,8 +2678,8 @@ def list_rows(
             timeout (Optional[float]):
                 The number of seconds to wait for the underlying HTTP transport
                 before using ``retry``.
-                If multiple requests are made under the hood, ``timeout`` is
-                interpreted as the approximate total time of **all** requests.
+                If multiple requests are made under the hood, ``timeout``
+                applies to each individual request.
 
         Returns:
             google.cloud.bigquery.table.RowIterator:
@@ -2711,11 +2704,7 @@ def list_rows(
         # No schema, but no selected_fields. Assume the developer wants all
         # columns, so get the table resource for them rather than failing.
         elif len(schema) == 0:
-            with TimeoutGuard(
-                timeout, timeout_error_type=concurrent.futures.TimeoutError
-            ) as guard:
-                table = self.get_table(table.reference, retry=retry, timeout=timeout)
-            timeout = guard.remaining_timeout
+            table = self.get_table(table.reference, retry=retry, timeout=timeout)
             schema = table.schema
 
         params = {}
diff --git a/google/cloud/bigquery/job.py b/google/cloud/bigquery/job.py
--- a/google/cloud/bigquery/job.py
+++ b/google/cloud/bigquery/job.py
@@ -26,7 +26,6 @@
 from six.moves import http_client
 
 import google.api_core.future.polling
-from google.auth.transport.requests import TimeoutGuard
 from google.cloud import exceptions
 from google.cloud.exceptions import NotFound
 from google.cloud.bigquery.dataset import Dataset
@@ -55,7 +54,6 @@
 _DONE_STATE = "DONE"
 _STOPPED_REASON = "stopped"
 _TIMEOUT_BUFFER_SECS = 0.1
-_SERVER_TIMEOUT_MARGIN_SECS = 1.0
 _CONTAINS_ORDER_BY = re.compile(r"ORDER\s+BY", re.IGNORECASE)
 
 _ERROR_REASON_TO_EXCEPTION = {
@@ -796,8 +794,8 @@ def result(self, retry=DEFAULT_RETRY, timeout=None):
             timeout (Optional[float]):
                 The number of seconds to wait for the underlying HTTP transport
                 before using ``retry``.
-                If multiple requests are made under the hood, ``timeout`` is
-                interpreted as the approximate total time of **all** requests.
+                If multiple requests are made under the hood, ``timeout``
+                applies to each individual request.
 
         Returns:
             _AsyncJob: This instance.
@@ -809,11 +807,7 @@ def result(self, retry=DEFAULT_RETRY, timeout=None):
                 if the job did not complete in the given timeout.
         """
         if self.state is None:
-            with TimeoutGuard(
-                timeout, timeout_error_type=concurrent.futures.TimeoutError
-            ) as guard:
-                self._begin(retry=retry, timeout=timeout)
-            timeout = guard.remaining_timeout
+            self._begin(retry=retry, timeout=timeout)
         # TODO: modify PollingFuture so it can pass a retry argument to done().
         return super(_AsyncJob, self).result(timeout=timeout)
 
@@ -2602,6 +2596,7 @@ def __init__(self, job_id, query, client, job_config=None):
         self._configuration = job_config
         self._query_results = None
         self._done_timeout = None
+        self._transport_timeout = None
 
     @property
     def allow_large_results(self):
@@ -3059,19 +3054,9 @@ def done(self, retry=DEFAULT_RETRY, timeout=None):
             self._done_timeout = max(0, self._done_timeout)
             timeout_ms = int(api_timeout * 1000)
 
-        # If the server-side processing timeout (timeout_ms) is specified and
-        # would be picked as the total request timeout, we want to add a small
-        # margin to it - we don't want to timeout the connection just as the
-        # server-side processing might have completed, but instead slightly
-        # after the server-side deadline.
-        # However, if `timeout` is specified, and is shorter than the adjusted
-        # server timeout, the former prevails.
-        if timeout_ms is not None and timeout_ms > 0:
-            server_timeout_with_margin = timeout_ms / 1000 + _SERVER_TIMEOUT_MARGIN_SECS
-            if timeout is not None:
-                timeout = min(server_timeout_with_margin, timeout)
-            else:
-                timeout = server_timeout_with_margin
+        # If an explicit timeout is not given, fall back to the transport timeout
+        # stored in _blocking_poll() in the process of polling for job completion.
+        transport_timeout = timeout if timeout is not None else self._transport_timeout
 
         # Do not refresh if the state is already done, as the job will not
         # change once complete.
@@ -3082,19 +3067,20 @@ def done(self, retry=DEFAULT_RETRY, timeout=None):
                 project=self.project,
                 timeout_ms=timeout_ms,
                 location=self.location,
-                timeout=timeout,
+                timeout=transport_timeout,
             )
 
             # Only reload the job once we know the query is complete.
             # This will ensure that fields such as the destination table are
             # correctly populated.
             if self._query_results.complete:
-                self.reload(retry=retry, timeout=timeout)
+                self.reload(retry=retry, timeout=transport_timeout)
 
         return self.state == _DONE_STATE
 
     def _blocking_poll(self, timeout=None):
         self._done_timeout = timeout
+        self._transport_timeout = timeout
         super(QueryJob, self)._blocking_poll(timeout=timeout)
 
     @staticmethod
@@ -3170,8 +3156,8 @@ def result(
             timeout (Optional[float]):
                 The number of seconds to wait for the underlying HTTP transport
                 before using ``retry``.
-                If multiple requests are made under the hood, ``timeout`` is
-                interpreted as the approximate total time of **all** requests.
+                If multiple requests are made under the hood, ``timeout``
+                applies to each individual request.
 
         Returns:
             google.cloud.bigquery.table.RowIterator:
@@ -3189,27 +3175,17 @@ def result(
                 If the job did not complete in the given timeout.
         """
         try:
-            guard = TimeoutGuard(
-                timeout, timeout_error_type=concurrent.futures.TimeoutError
-            )
-            with guard:
-                super(QueryJob, self).result(retry=retry, timeout=timeout)
-            timeout = guard.remaining_timeout
+            super(QueryJob, self).result(retry=retry, timeout=timeout)
 
             # Return an iterator instead of returning the job.
             if not self._query_results:
-                guard = TimeoutGuard(
-                    timeout, timeout_error_type=concurrent.futures.TimeoutError
+                self._query_results = self._client._get_query_results(
+                    self.job_id,
+                    retry,
+                    project=self.project,
+                    location=self.location,
+                    timeout=timeout,
                 )
-                with guard:
-                    self._query_results = self._client._get_query_results(
-                        self.job_id,
-                        retry,
-                        project=self.project,
-                        location=self.location,
-                        timeout=timeout,
-                    )
-                timeout = guard.remaining_timeout
         except exceptions.GoogleCloudError as exc:
             exc.message += self._format_for_exception(self.query, self.job_id)
             exc.query_job = self


Test patch:
diff --git a/tests/unit/test_client.py b/tests/unit/test_client.py
--- a/tests/unit/test_client.py
+++ b/tests/unit/test_client.py
@@ -24,7 +24,6 @@
 import unittest
 import warnings
 
-import freezegun
 import mock
 import requests
 import six
@@ -5496,43 +5495,6 @@ def test_list_partitions_with_string_id(self):
 
         self.assertEqual(len(partition_list), 0)
 
-    def test_list_partitions_splitting_timout_between_requests(self):
-        from google.cloud.bigquery.table import Table
-
-        row_count = 2
-        meta_info = _make_list_partitons_meta_info(
-            self.PROJECT, self.DS_ID, self.TABLE_ID, row_count
-        )
-
-        data = {
-            "totalRows": str(row_count),
-            "rows": [{"f": [{"v": "20180101"}]}, {"f": [{"v": "20180102"}]}],
-        }
-        creds = _make_credentials()
-        http = object()
-        client = self._make_one(project=self.PROJECT, credentials=creds, _http=http)
-        client._connection = make_connection(meta_info, data)
-        table = Table(self.TABLE_REF)
-
-        with freezegun.freeze_time("2019-01-01 00:00:00", tick=False) as frozen_time:
-
-            def delayed_get_table(*args, **kwargs):
-                frozen_time.tick(delta=1.4)
-                return orig_get_table(*args, **kwargs)
-
-            orig_get_table = client.get_table
-            client.get_table = mock.Mock(side_effect=delayed_get_table)
-
-            client.list_partitions(table, timeout=5.0)
-
-        client.get_table.assert_called_once()
-        _, kwargs = client.get_table.call_args
-        self.assertEqual(kwargs.get("timeout"), 5.0)
-
-        client._connection.api_request.assert_called()
-        _, kwargs = client._connection.api_request.call_args
-        self.assertAlmostEqual(kwargs.get("timeout"), 3.6, places=5)
-
     def test_list_rows(self):
         import datetime
         from google.cloud._helpers import UTC
@@ -5918,46 +5880,6 @@ def test_list_rows_with_missing_schema(self):
             self.assertEqual(rows[1].age, 31, msg=repr(table))
             self.assertIsNone(rows[2].age, msg=repr(table))
 
-    def test_list_rows_splitting_timout_between_requests(self):
-        from google.cloud.bigquery.schema import SchemaField
-        from google.cloud.bigquery.table import Table
-
-        response = {"totalRows": "0", "rows": []}
-        creds = _make_credentials()
-        http = object()
-        client = self._make_one(project=self.PROJECT, credentials=creds, _http=http)
-        client._connection = make_connection(response, response)
-
-        table = Table(
-            self.TABLE_REF, schema=[SchemaField("field_x", "INTEGER", mode="NULLABLE")]
-        )
-
-        with freezegun.freeze_time("1970-01-01 00:00:00", tick=False) as frozen_time:
-
-            def delayed_get_table(*args, **kwargs):
-                frozen_time.tick(delta=1.4)
-                return table
-
-            client.get_table = mock.Mock(side_effect=delayed_get_table)
-
-            rows_iter = client.list_rows(
-                "{}.{}.{}".format(
-                    self.TABLE_REF.project,
-                    self.TABLE_REF.dataset_id,
-                    self.TABLE_REF.table_id,
-                ),
-                timeout=5.0,
-            )
-            six.next(rows_iter.pages)
-
-        client.get_table.assert_called_once()
-        _, kwargs = client.get_table.call_args
-        self.assertEqual(kwargs.get("timeout"), 5.0)
-
-        client._connection.api_request.assert_called_once()
-        _, kwargs = client._connection.api_request.call_args
-        self.assertAlmostEqual(kwargs.get("timeout"), 3.6)
-
     def test_list_rows_error(self):
         creds = _make_credentials()
         http = object()
diff --git a/tests/unit/test_job.py b/tests/unit/test_job.py
--- a/tests/unit/test_job.py
+++ b/tests/unit/test_job.py
@@ -994,24 +994,6 @@ def test_result_explicit_w_state(self, result):
         begin.assert_not_called()
         result.assert_called_once_with(timeout=timeout)
 
-    @mock.patch("google.api_core.future.polling.PollingFuture.result")
-    def test_result_splitting_timout_between_requests(self, result):
-        client = _make_client(project=self.PROJECT)
-        job = self._make_one(self.JOB_ID, client)
-        begin = job._begin = mock.Mock()
-        retry = mock.Mock()
-
-        with freezegun.freeze_time("1970-01-01 00:00:00", tick=False) as frozen_time:
-
-            def delayed_begin(*args, **kwargs):
-                frozen_time.tick(delta=0.3)
-
-            begin.side_effect = delayed_begin
-            job.result(retry=retry, timeout=1.0)
-
-        begin.assert_called_once_with(retry=retry, timeout=1.0)
-        result.assert_called_once_with(timeout=0.7)
-
     def test_cancelled_wo_error_result(self):
         client = _make_client(project=self.PROJECT)
         job = self._make_one(self.JOB_ID, client)
@@ -4011,33 +3993,6 @@ def test_done_w_timeout(self):
         call_args = fake_reload.call_args
         self.assertEqual(call_args.kwargs.get("timeout"), 42)
 
-    def test_done_w_timeout_and_shorter_internal_api_timeout(self):
-        from google.cloud.bigquery.job import _TIMEOUT_BUFFER_SECS
-        from google.cloud.bigquery.job import _SERVER_TIMEOUT_MARGIN_SECS
-
-        client = _make_client(project=self.PROJECT)
-        resource = self._make_resource(ended=False)
-        job = self._get_target_class().from_api_repr(resource, client)
-        job._done_timeout = 8.8
-
-        with mock.patch.object(
-            client, "_get_query_results"
-        ) as fake_get_results, mock.patch.object(job, "reload") as fake_reload:
-            job.done(timeout=42)
-
-        # The expected timeout used is the job's own done_timeout minus a
-        # fixed amount (bigquery.job._TIMEOUT_BUFFER_SECS) increased by the
-        # safety margin on top of server-side processing timeout - that's
-        # because that final number is smaller than the given timeout (42 seconds).
-        expected_timeout = 8.8 - _TIMEOUT_BUFFER_SECS + _SERVER_TIMEOUT_MARGIN_SECS
-
-        fake_get_results.assert_called_once()
-        call_args = fake_get_results.call_args
-        self.assertAlmostEqual(call_args.kwargs.get("timeout"), expected_timeout)
-
-        call_args = fake_reload.call_args
-        self.assertAlmostEqual(call_args.kwargs.get("timeout"), expected_timeout)
-
     def test_done_w_timeout_and_longer_internal_api_timeout(self):
         client = _make_client(project=self.PROJECT)
         resource = self._make_resource(ended=False)
@@ -4623,49 +4578,6 @@ def test_result_w_timeout(self):
         self.assertEqual(query_request[1]["query_params"]["timeoutMs"], 900)
         self.assertEqual(reload_request[1]["method"], "GET")
 
-    @mock.patch("google.api_core.future.polling.PollingFuture.result")
-    def test_result_splitting_timout_between_requests(self, polling_result):
-        begun_resource = self._make_resource()
-        query_resource = {
-            "jobComplete": True,
-            "jobReference": {"projectId": self.PROJECT, "jobId": self.JOB_ID},
-            "schema": {"fields": [{"name": "col1", "type": "STRING"}]},
-            "totalRows": "5",
-        }
-        done_resource = copy.deepcopy(begun_resource)
-        done_resource["status"] = {"state": "DONE"}
-
-        connection = _make_connection(begun_resource, query_resource, done_resource)
-        client = _make_client(project=self.PROJECT, connection=connection)
-        job = self._make_one(self.JOB_ID, self.QUERY, client)
-
-        client.list_rows = mock.Mock()
-
-        with freezegun.freeze_time("1970-01-01 00:00:00", tick=False) as frozen_time:
-
-            def delayed_result(*args, **kwargs):
-                frozen_time.tick(delta=0.8)
-
-            polling_result.side_effect = delayed_result
-
-            def delayed_get_results(*args, **kwargs):
-                frozen_time.tick(delta=0.5)
-                return orig_get_results(*args, **kwargs)
-
-            orig_get_results = client._get_query_results
-            client._get_query_results = mock.Mock(side_effect=delayed_get_results)
-            job.result(timeout=2.0)
-
-        polling_result.assert_called_once_with(timeout=2.0)
-
-        client._get_query_results.assert_called_once()
-        _, kwargs = client._get_query_results.call_args
-        self.assertAlmostEqual(kwargs.get("timeout"), 1.2)
-
-        client.list_rows.assert_called_once()
-        _, kwargs = client.list_rows.call_args
-        self.assertAlmostEqual(kwargs.get("timeout"), 0.7)
-
     def test_result_w_page_size(self):
         # Arrange
         query_results_resource = {