Repository: googleapis/google-cloud-python

Issue:
BigQuery Job objects not pickle-able
@tseaver suggested I start a new issue after adding to a closed issue: https://github.com/GoogleCloudPlatform/google-cloud-python/issues/3191

BigQuery Job Objects are not pickle-able:

```

In [16]: job = client.copy_table(source_table,source_table)

In [17]: job
Out[17]: <google.cloud.bigquery.job.CopyJob at 0x1163a3320>

In [18]: pickle.dumps(job)
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-18-0c1255e2d696> in <module>()
----> 1 pickle.dumps(job)

AttributeError: Can't pickle local object 'if_exception_type.<locals>.if_exception_type_predicate'
```

I'm guessing that's because it represents some sort of future? IMO it would be fine if `__getstate__` deleted the polling and let a reference that could be checked.

@tseaver suggests: 

> As a workaround, you could pickle the result of job._to_api_repr, and then reconstitute it using client.job_from_resource.

which works. This could also be done by `__getstate__` itself.

Thanks


Discussion:
@tseaver CopyJob doesnt seem to have to_api_repr. Can you suggest which Class type does the job you are referring to belongs to ?
@c1dc-candidate-22 Hmm, looking at the code, the method I had in mind isn't called `to_api_repr`:  it is [`CopyJob._build_resource`](https://github.com/googleapis/google-cloud-python/blob/05b2e3744bd10d20ab4ca498a7abad0dc1e9ee60/bigquery/google/cloud/bigquery/job.py#L1546-L1570).  ISTM that renaming it `to_api_repr` would be reasonable (of course leaving behind an alias for backward compatibility).  @shollyman, @tswast WDYT?
This is where I fully admit I have not fully internalized this veneer yet, so I'm happy to defer to others.  `_build_resource` seems to revolve around ensuring a viable resource suitable for calling the backend for insertion (e.g. solely the configuration part of a job representation), whereas the goal here is the ability to serialize the whole representation of job: configuration, jobreference, statistics etc.

Exposing the full representation seems fine, but its not clear that aliasing is the right thing to do here without further reading on my part. Perhaps @tswast can offer more insight?

Job is the one set of resources that we didn't fully refactor during the 1.0 rewrite to follow the `to_api_repr` / `from_api_repr` pattern. I agree that having these methods is desirable.

It'll be a little bit tricky since Job mutates itself with the private `reload()` function. It'd be good to remove some of the fancy logic that Job has for building it's internal state and use the `_properties` dictionary directly, as we do for other resources.
Maybe the reason for the error is the data format used is not Python-specific? 
Also, don't forget to "import pickle" and "import client".
@tswast If `_build_resource` as it stands today were renamed `to_api_repr`, that would satisfy the OP's usecase.
Yes, it looks like `_build_resource` has no side effects. We can rename to `to_api_repr`.

Resolving patch:
diff --git a/bigquery/google/cloud/bigquery/client.py b/bigquery/google/cloud/bigquery/client.py
--- a/bigquery/google/cloud/bigquery/client.py
+++ b/bigquery/google/cloud/bigquery/client.py
@@ -949,7 +949,7 @@ def load_table_from_file(
 
         job_ref = job._JobReference(job_id, project=project, location=location)
         load_job = job.LoadJob(job_ref, None, destination, self, job_config)
-        job_resource = load_job._build_resource()
+        job_resource = load_job.to_api_repr()
 
         if rewind:
             file_obj.seek(0, os.SEEK_SET)
diff --git a/bigquery/google/cloud/bigquery/job.py b/bigquery/google/cloud/bigquery/job.py
--- a/bigquery/google/cloud/bigquery/job.py
+++ b/bigquery/google/cloud/bigquery/job.py
@@ -519,10 +519,12 @@ def _get_resource_config(cls, resource):
                            '["configuration"]["%s"]' % cls._JOB_TYPE)
         return job_id, resource['configuration']
 
-    def _build_resource(self):
-        """Helper:  Generate a resource for :meth:`_begin`."""
+    def to_api_repr(self):
+        """Generate a resource for the job."""
         raise NotImplementedError("Abstract")
 
+    _build_resource = to_api_repr  # backward-compatibility alias
+
     def _begin(self, client=None, retry=DEFAULT_RETRY):
         """API call:  begin the job via a POST request
 
@@ -549,7 +551,7 @@ def _begin(self, client=None, retry=DEFAULT_RETRY):
         # job has an ID.
         api_response = client._call_api(
             retry,
-            method='POST', path=path, data=self._build_resource())
+            method='POST', path=path, data=self.to_api_repr())
         self._set_properties(api_response)
 
     def exists(self, client=None, retry=DEFAULT_RETRY):
@@ -1367,7 +1369,7 @@ def output_rows(self):
         if statistics is not None:
             return int(statistics['load']['outputRows'])
 
-    def _build_resource(self):
+    def to_api_repr(self):
         """Generate a resource for :meth:`_begin`."""
         configuration = self._configuration.to_api_repr()
         if self.source_uris is not None:
@@ -1543,7 +1545,7 @@ def destination_encryption_configuration(self):
         """
         return self._configuration.destination_encryption_configuration
 
-    def _build_resource(self):
+    def to_api_repr(self):
         """Generate a resource for :meth:`_begin`."""
 
         source_refs = [{
@@ -1761,7 +1763,7 @@ def destination_uri_file_counts(self):
             return [int(count) for count in counts]
         return None
 
-    def _build_resource(self):
+    def to_api_repr(self):
         """Generate a resource for :meth:`_begin`."""
 
         source_ref = {
@@ -2367,7 +2369,7 @@ def schema_update_options(self):
         """
         return self._configuration.schema_update_options
 
-    def _build_resource(self):
+    def to_api_repr(self):
         """Generate a resource for :meth:`_begin`."""
         configuration = self._configuration.to_api_repr()
 


Test patch:
diff --git a/bigquery/tests/unit/test_client.py b/bigquery/tests/unit/test_client.py
--- a/bigquery/tests/unit/test_client.py
+++ b/bigquery/tests/unit/test_client.py
@@ -2214,7 +2214,7 @@ def _initiate_resumable_upload_helper(self, num_retries=None):
         config = LoadJobConfig()
         config.source_format = SourceFormat.CSV
         job = LoadJob(None, None, self.TABLE_REF, client, job_config=config)
-        metadata = job._build_resource()
+        metadata = job.to_api_repr()
         upload, transport = client._initiate_resumable_upload(
             stream, metadata, num_retries)
 
@@ -2279,7 +2279,7 @@ def _do_multipart_upload_success_helper(
         config = LoadJobConfig()
         config.source_format = SourceFormat.CSV
         job = LoadJob(None, None, self.TABLE_REF, client, job_config=config)
-        metadata = job._build_resource()
+        metadata = job.to_api_repr()
         size = len(data)
         response = client._do_multipart_upload(
             stream, metadata, size, num_retries)
diff --git a/bigquery/tests/unit/test_job.py b/bigquery/tests/unit/test_job.py
--- a/bigquery/tests/unit/test_job.py
+++ b/bigquery/tests/unit/test_job.py
@@ -522,6 +522,12 @@ def test__build_resource(self):
         with self.assertRaises(NotImplementedError):
             job._build_resource()
 
+    def test_to_api_repr(self):
+        client = _make_client(project=self.PROJECT)
+        job = self._make_one(self.JOB_ID, client)
+        with self.assertRaises(NotImplementedError):
+            job.to_api_repr()
+
     def test__begin_already(self):
         job = self._set_properties_job()
         job._properties['status'] = {'state': 'WHATEVER'}
@@ -543,7 +549,7 @@ def test__begin_defaults(self):
             }
         }
         job = self._set_properties_job()
-        builder = job._build_resource = mock.Mock()
+        builder = job.to_api_repr = mock.Mock()
         builder.return_value = resource
         call_api = job._client._call_api = mock.Mock()
         call_api.return_value = resource
@@ -573,7 +579,7 @@ def test__begin_explicit(self):
             }
         }
         job = self._set_properties_job()
-        builder = job._build_resource = mock.Mock()
+        builder = job.to_api_repr = mock.Mock()
         builder.return_value = resource
         client = _make_client(project=other_project)
         call_api = client._call_api = mock.Mock()