Repository: googleapis/google-cloud-python

Issue:
BigQuery: jobs created with run_async_query() and run with begin() ignores flag dry_run=True
The aim is to be able to dry run the queries to check how much it would cost us before we actually proceed with it. Hopefully we will be able to use it as a safety mechanism. 

I hope I am not misunderstanding the availability of the `dry_run` parameter as [listed on the docs](https://googlecloudplatform.github.io/google-cloud-python/stable/bigquery-job.html#google.cloud.bigquery.job.QueryJob.dry_run). I have set it, and it appears on the configs, **but it runs anyway**. 


1. Ubuntu 14.04

2. Python 2.7.6

3. google-cloud-python version
```
google-api-python-client==1.5.5
google-auth==0.5.0
google-auth-httplib2==0.0.2
google-cloud-bigquery==0.22.0
google-cloud-core==0.22.1
google-cloud-storage==0.22.0
googleapis-common-protos==1.5.0
```

4. Stacktrace if available
NIL

5. Steps to reproduce
- create job with run_async_query()
- set job.dry_run = True
- run it with job.begin()
- examine job when it is done

6. Code example
```
job = client.run_async_query(JOB_NAME, QUERY_STRING)
job.dry_run = True
job.begin()
print job.__dict__
```


Discussion:
@tseaver We should probably add `__slots__` at this point. There have just been too many of these is it accepted / isn't it accepted issues.
@dhermes This isn't that:  we actually do have a `dry_run` property [defined](https://github.com/GoogleCloudPlatform/google-cloud-python/blob/7e1632598365af1c03709bb5f5ff29f4d8a98847/bigquery/google/cloud/bigquery/job.py#L983), and we [copy it to the job configuration if set](https://github.com/GoogleCloudPlatform/google-cloud-python/blob/7e1632598365af1c03709bb5f5ff29f4d8a98847/bigquery/google/cloud/bigquery/job.py#L1027-L1028).
Interesting. What do you think is the issue here?
@dhermes We might have a bug on our side (e.g., copying to the wrong part of the job config resource, or with the wrong name), or the back-end might be ignoring the `dryRun` flag.
Easy enough to add a `job.dry_run = True` before calling `.begin()` in a system test and adding a handy-dandy `import pdb; pdb.set_trace()` when `api_request()` actually sends off a payload?
Hi, yes I am referring to an issue like @tseaver has pointed out, that the dry_run configuration was accepted (was mentioned and fixed in an earlier issue, I had checked before posting). I attempted to read and trace the code to where the configurations were being passed around, but unfortunately I don't think I'm quite familiar enough.

I tried using `pdb` as suggested, and I got to a point where I saw this:

`
/home/dev-sean/bq_ops/lib/python2.7/site-packages/google/cloud/bigquery/job.py(1080)_build_resource()->{'configuration': {'query': {'dryRun': True, 'query': 'SELECT * FRO...x.__TABLES__]'}}, 'jobReference': {'jobId': 'debug-dry_ru...-880691b3d08b', 'projectId': u'test-b...ery-1043'}}
`

According to the documentation here:
https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.query.dryRun

The dryRun property should be parked under `configuration` and not under `query`.

Is this perhaps what @dhermes is hoping to find?
@sean185 Almost certainly what I was hoping for.
@dhermes @tseaver 
I don't know if this is relevant, but on the docs for Jobs > QueryJob
https://googlecloudplatform.github.io/google-cloud-python/stable/bigquery-job.html#google.cloud.bigquery.job.QueryJob.dry_run

I found that it mentioned this link for the documentation regarding dryRun,
https://cloud.google.com/bigquery/docs/reference/v2/jobs#configuration.query.dryRun

when the hyperlink should have in fact been 
https://cloud.google.com/bigquery/docs/reference/v2/jobs#configuration.dryRun

as per the nesting specified in the docs. 
@dhermes I wonder if we've been gaslighted:  note the language describing the option:

```
[Optional] If set, don't actually run this job. A valid query will return a mostly empty response with some processing statistics, while an invalid query will return the same error it would if it wasn't a dry run. Behavior of non-query jobs is undefined.
```

Five will get you ten they "promoted" that option up from the `query` section without telling anybody, or rev'ing the API.
@sean185 Thanks for digging into that!  We need to update to write the flag at the `config` level, rather than `config`/`query`.
@tseaver no problem, just glad that it wasn't a case of me running blind.

So just to clarify, it is [this chunk of code](https://github.com/GoogleCloudPlatform/google-cloud-python/blob/7e1632598365af1c03709bb5f5ff29f4d8a98847/bigquery/google/cloud/bigquery/job.py#L1067-L1084), the `_build_resource` function that places the `dry_run` variable too deep, correct?
@sean185 Yes, it calls (eventually) `_populate_config_booleans`, which does the actual work.
@tseaver ah okay, got it! Will be looking forward to how this issue will be managed, thank you guys for the great work!
@tseaver @dhermes have one of you guys started on this already? If not I can make a PR.
@daspecster Go ahead!

Resolving patch:
diff --git a/bigquery/google/cloud/bigquery/job.py b/bigquery/google/cloud/bigquery/job.py
--- a/bigquery/google/cloud/bigquery/job.py
+++ b/bigquery/google/cloud/bigquery/job.py
@@ -983,7 +983,7 @@ def __init__(self, name, query, client,
     dry_run = _TypedProperty('dry_run', bool)
     """See:
     https://cloud.google.com/bigquery/docs/\
-    reference/v2/jobs#configuration.query.dryRun
+    reference/rest/v2/jobs#configuration.dryRun
     """
 
     write_disposition = WriteDisposition('write_disposition')
@@ -1024,8 +1024,6 @@ def _populate_config_resource_booleans(self, configuration):
             configuration['useQueryCache'] = self.use_query_cache
         if self.use_legacy_sql is not None:
             configuration['useLegacySql'] = self.use_legacy_sql
-        if self.dry_run is not None:
-            configuration['dryRun'] = self.dry_run
 
     def _populate_config_resource(self, configuration):
         """Helper for _build_resource: copy config properties to resource"""
@@ -1078,6 +1076,10 @@ def _build_resource(self):
                 },
             },
         }
+
+        if self.dry_run is not None:
+            resource['configuration']['dryRun'] = self.dry_run
+
         configuration = resource['configuration'][self._JOB_TYPE]
         self._populate_config_resource(configuration)
 


Test patch:
diff --git a/bigquery/unit_tests/test_job.py b/bigquery/unit_tests/test_job.py
--- a/bigquery/unit_tests/test_job.py
+++ b/bigquery/unit_tests/test_job.py
@@ -1331,11 +1331,6 @@ def _verifyBooleanResourceProperties(self, job, config):
                              config['useLegacySql'])
         else:
             self.assertIsNone(job.use_legacy_sql)
-        if 'dryRun' in config:
-            self.assertEqual(job.dry_run,
-                             config['dryRun'])
-        else:
-            self.assertIsNone(job.dry_run)
 
     def _verifyIntegerResourceProperties(self, job, config):
         if 'maximumBillingTier' in config:
@@ -1366,48 +1361,58 @@ def _verifyQueryParameters(self, job, config):
         for found, expected in zip(job.query_parameters, query_parameters):
             self.assertEqual(found.to_api_repr(), expected)
 
+    def _verify_configuration_properties(self, job, configuration):
+        if 'dryRun' in configuration:
+            self.assertEqual(job.dry_run,
+                             configuration['dryRun'])
+        else:
+            self.assertIsNone(job.dry_run)
+
     def _verifyResourceProperties(self, job, resource):
         self._verifyReadonlyResourceProperties(job, resource)
 
-        config = resource.get('configuration', {}).get('query')
-        self._verifyBooleanResourceProperties(job, config)
-        self._verifyIntegerResourceProperties(job, config)
-        self._verify_udf_resources(job, config)
-        self._verifyQueryParameters(job, config)
+        configuration = resource.get('configuration', {})
+        self._verify_configuration_properties(job, configuration)
 
-        self.assertEqual(job.query, config['query'])
-        if 'createDisposition' in config:
+        query_config = resource.get('configuration', {}).get('query')
+        self._verifyBooleanResourceProperties(job, query_config)
+        self._verifyIntegerResourceProperties(job, query_config)
+        self._verify_udf_resources(job, query_config)
+        self._verifyQueryParameters(job, query_config)
+
+        self.assertEqual(job.query, query_config['query'])
+        if 'createDisposition' in query_config:
             self.assertEqual(job.create_disposition,
-                             config['createDisposition'])
+                             query_config['createDisposition'])
         else:
             self.assertIsNone(job.create_disposition)
-        if 'defaultDataset' in config:
+        if 'defaultDataset' in query_config:
             dataset = job.default_dataset
             ds_ref = {
                 'projectId': dataset.project,
                 'datasetId': dataset.name,
             }
-            self.assertEqual(ds_ref, config['defaultDataset'])
+            self.assertEqual(ds_ref, query_config['defaultDataset'])
         else:
             self.assertIsNone(job.default_dataset)
-        if 'destinationTable' in config:
+        if 'destinationTable' in query_config:
             table = job.destination
             tb_ref = {
                 'projectId': table.project,
                 'datasetId': table.dataset_name,
                 'tableId': table.name
             }
-            self.assertEqual(tb_ref, config['destinationTable'])
+            self.assertEqual(tb_ref, query_config['destinationTable'])
         else:
             self.assertIsNone(job.destination)
-        if 'priority' in config:
+        if 'priority' in query_config:
             self.assertEqual(job.priority,
-                             config['priority'])
+                             query_config['priority'])
         else:
             self.assertIsNone(job.priority)
-        if 'writeDisposition' in config:
+        if 'writeDisposition' in query_config:
             self.assertEqual(job.write_disposition,
-                             config['writeDisposition'])
+                             query_config['writeDisposition'])
         else:
             self.assertIsNone(job.write_disposition)
 
@@ -1575,7 +1580,6 @@ def test_begin_w_alternate_client(self):
             'priority': 'INTERACTIVE',
             'useQueryCache': True,
             'useLegacySql': True,
-            'dryRun': True,
             'writeDisposition': 'WRITE_TRUNCATE',
             'maximumBillingTier': 4,
             'maximumBytesBilled': 123456
@@ -1599,6 +1603,7 @@ def test_begin_w_alternate_client(self):
         job.use_query_cache = True
         job.use_legacy_sql = True
         job.dry_run = True
+        RESOURCE['configuration']['dryRun'] = True
         job.write_disposition = 'WRITE_TRUNCATE'
         job.maximum_billing_tier = 4
         job.maximum_bytes_billed = 123456
@@ -1616,6 +1621,7 @@ def test_begin_w_alternate_client(self):
                 'jobId': self.JOB_NAME,
             },
             'configuration': {
+                'dryRun': True,
                 'query': QUERY_CONFIGURATION,
             },
         }
@@ -1775,6 +1781,41 @@ def test_begin_w_positional_query_parameter(self):
         self.assertEqual(req['data'], SENT)
         self._verifyResourceProperties(job, RESOURCE)
 
+    def test_dry_run_query(self):
+        PATH = '/projects/%s/jobs' % (self.PROJECT,)
+        RESOURCE = self._makeResource()
+        # Ensure None for missing server-set props
+        del RESOURCE['statistics']['creationTime']
+        del RESOURCE['etag']
+        del RESOURCE['selfLink']
+        del RESOURCE['user_email']
+        conn = _Connection(RESOURCE)
+        client = _Client(project=self.PROJECT, connection=conn)
+        job = self._make_one(self.JOB_NAME, self.QUERY, client)
+        job.dry_run = True
+        RESOURCE['configuration']['dryRun'] = True
+
+        job.begin()
+        self.assertEqual(job.udf_resources, [])
+        self.assertEqual(len(conn._requested), 1)
+        req = conn._requested[0]
+        self.assertEqual(req['method'], 'POST')
+        self.assertEqual(req['path'], PATH)
+        SENT = {
+            'jobReference': {
+                'projectId': self.PROJECT,
+                'jobId': self.JOB_NAME,
+            },
+            'configuration': {
+                'query': {
+                    'query': self.QUERY
+                },
+                'dryRun': True,
+            },
+        }
+        self.assertEqual(req['data'], SENT)
+        self._verifyResourceProperties(job, RESOURCE)
+
     def test_exists_miss_w_bound_client(self):
         PATH = '/projects/%s/jobs/%s' % (self.PROJECT, self.JOB_NAME)
         conn = _Connection()