Repository: googleapis/python-bigquery

Issue:
[Breaking Change] Don't remove destinationTable attribute in `create_job`
# What happened:

I run a job with a query and a destination table and the table is not generated from the file.
```python
bq_client.create_job({
    'query': {
        'query': rendered_query,
        'destinationTable': {
            'projectId': GCP_PROJECT,
            'datasetId': DATASET_ID,
            'tableId': TRANSFORMED_TABLE_ID
        },
        'writeDisposition': 'WRITE_TRUNCATE'
})
```

My code executes successfully and produces the following job in the console
![image](https://user-images.githubusercontent.com/3038962/105621129-ab321b80-5e04-11eb-820e-a09631a8a379.png)



# How to fix it
There is a line where the `destinationTable` is removed from the job config. Removing the line that deletes the property in the job configuration will do.
https://github.com/googleapis/python-bigquery/blob/530e1e8d8fe8939e914a78ff1b220907c1b87af7/google/cloud/bigquery/client.py#L1729

Can anyone explain why is the `destinationTable` property removed?

In the meanwhile I'm opening a PR


Discussion:
From the [description](https://github.com/googleapis/python-bigquery/issues/14) of the ticket when this was added:

> For example, BigQuery automatically populates a destination table if not set. In this case, the destination table should not be part of the retried request. In my opinion, this kind of logic is outside the scope of the client libraries, as it's not clear when destination table needs to be cleared just from the job resource.

While the client library always removes the destination table, that might not be the correct in all cases, but then again, it might be difficult to determine in every case what action to take.

@tswast Do you remember the details and what the backend expects? Is it possible/feasible for the client to be in sync with the backend without duplicating the latter's logic?
> @tswast Do you remember the details and what the backend expects? Is it possible/feasible for the client to be in sync with the backend without duplicating the latter's logic?

The main problem is permissions. When the destination table is automatically populated, it creates a table in a special dataset where data automatically expires and the customer is not charged for storage. The customer does not have permission to write directly to such a table.

For example, I just ran a query and it wrote the results to the temporary table `	
swast-scratch:_63cfa399614a54153cc386c27d6c0c6fdb249f9e.anonb14b7884b8147f85ae0731c3a6f8d0026af97ab`

In every case I've seen, the temporary table `dataset_id` begins with `_` and the table name begins with `anon`.

We could possibly only strip the destination table in these cases. There's still a risk of false positives, but it should be much less likely.
OK thanks, this sounds like a better heuristic. If it turns out that it doesn't work well enough in practice, it should be easy to modify/remove, if necessary.
I might be failing to fully understand why the python API/library works like it does because I lack this context you are talking about.

What I want to achieve is the same functionality as expected and provided by other ways I interact with bigquery:
* [REST API and gRPC API][api]
* [NodeJs library][node]
* [Golang library][go]
* Airflow with the [BigQueryInsertJobOperator][airflow] with the same method`createJob`

@tswast can you help me understand this use-case with the temp table? It looks to me like a client logic. But again, I think I'm failing to fully grasp the example.

I can seamlessly migrate mi code between several languages simply moving the job configuration (dictionary/json/struct) and I don't understand why the python library is an exception.

I don't want to be rude by any means. And I respect your work, which is fantastic.

[node]: https://github.com/googleapis/nodejs-bigquery/blob/9481a2b90700739a0f36c31ae9a3f1efa0c8e0c1/src/bigquery.ts#L1423
[api]: https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs/insert
[airflow]: https://github.com/apache/airflow/blob/master/airflow/providers/google/cloud/operators/bigquery.py#L1933
[go]: https://github.com/googleapis/google-cloud-go/blob/master/bigquery/bigquery.go#L84
@dinigo Full context is here: https://github.com/googleapis/google-cloud-python/issues/5555 and here: https://github.com/googleapis/python-bigquery/issues/14

You're probably right that the correct choice back then would have been to keep `create_job` simple and perhaps had a helper to "reset" any server-side job configuration so that it can retried. Unfortunately we did not make that choice, so we need to avoid breaking that "retry this job, please" use-case.
Thanks @tswast ! I get it now. Also You have to keep the API backwards compatible ;)

No problem. We will use a fork and rebase regularly since we need and expect this funcionality. 

I would like though to have this feature integrated at some point in the future. If you could tag it as such. Thanks again
@jimfulton I think if we get retryable queries working in https://github.com/googleapis/python-bigquery/issues/539, we should be able to remove this odd behavior of `create_job`.

Also, perhaps if `create_job` is called with any of our `*Job` classes, it uses the original request that we cache rather than trying to figure out which properties on the response resource we can send back?
Some notes:
- The original use case that drove adding this was retry: https://github.com/googleapis/google-cloud-python/issues/5555, but https://github.com/googleapis/python-bigquery/pull/837 should remove the need to retry jobs.
- Therefore, after #837 lands, we'll stop removing destination from the config used for query jobs.
- This is blocked pending #837.

Resolving patch:
diff --git a/google/cloud/bigquery/client.py b/google/cloud/bigquery/client.py
--- a/google/cloud/bigquery/client.py
+++ b/google/cloud/bigquery/client.py
@@ -49,7 +49,6 @@
     DEFAULT_CLIENT_INFO as DEFAULT_BQSTORAGE_CLIENT_INFO,
 )
 
-from google.cloud.bigquery._helpers import _del_sub_prop
 from google.cloud.bigquery._helpers import _get_sub_prop
 from google.cloud.bigquery._helpers import _record_field_to_json
 from google.cloud.bigquery._helpers import _str_or_none
@@ -1937,7 +1936,6 @@ def create_job(
             )
         elif "query" in job_config:
             copy_config = copy.deepcopy(job_config)
-            _del_sub_prop(copy_config, ["query", "destinationTable"])
             query_job_config = google.cloud.bigquery.job.QueryJobConfig.from_api_repr(
                 copy_config
             )


Test patch:
diff --git a/tests/unit/test_client.py b/tests/unit/test_client.py
--- a/tests/unit/test_client.py
+++ b/tests/unit/test_client.py
@@ -2594,8 +2594,6 @@ def test_delete_table_w_not_found_ok_true(self):
         conn.api_request.assert_called_with(method="DELETE", path=path, timeout=None)
 
     def _create_job_helper(self, job_config):
-        from google.cloud.bigquery import _helpers
-
         creds = _make_credentials()
         http = object()
         client = self._make_one(project=self.PROJECT, credentials=creds, _http=http)
@@ -2606,8 +2604,6 @@ def _create_job_helper(self, job_config):
         }
         conn = client._connection = make_connection(RESOURCE)
         client.create_job(job_config=job_config)
-        if "query" in job_config:
-            _helpers._del_sub_prop(job_config, ["query", "destinationTable"])
 
         conn.api_request.assert_called_once_with(
             method="POST",
@@ -2732,7 +2728,7 @@ def test_create_job_query_config_w_rateLimitExceeded_error(self):
         }
         data_without_destination = {
             "jobReference": {"projectId": self.PROJECT, "jobId": mock.ANY},
-            "configuration": {"query": {"query": query, "useLegacySql": False}},
+            "configuration": configuration,
         }
 
         creds = _make_credentials()