Repository: kubeflow/pipelines

Issue:
BrokenPipe for ml engine train component
### What steps did you take:
ML engine train component throws BrokenPipe Error after around 30 minutes of continuous training. This is not a one-off situation but jobs keep failing.
### What happened:

### What did you expect to happen:

### Environment:
<!-- Please fill in those that seem relevant. -->

How did you deploy Kubeflow Pipelines (KFP)?
Kubeflow Pipelines was deployed using gcp "ai platform pipelines" tool.
Kubeflow Pipelines 1.0.4
Kubernetes: 1.17.14-gke.1600

```
ML_ENGINE_TRAIN_OP = comp.load_component_from_url(
    "https://raw.githubusercontent.com/kubeflow/pipelines/1.3.0/components/gcp/ml_engine/train/component.yaml"
)
```
<!-- If you are not sure, here's [an introduction of all options](https://www.kubeflow.org/docs/pipelines/installation/overview/). -->

KFP version: 1.0.4

KFP SDK version: 1.3.0


### Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

/kind bug
<!-- Please include labels by uncommenting them to help us better triage issues, choose from the following -->
<!--
// /area frontend
// /area backend
// /area sdk
// /area testing
// /area engprod
-->
![image](https://user-images.githubusercontent.com/32985207/106754349-29749600-662d-11eb-94db-c9e30ef639fe.png)



Discussion:
/cc @chensun @neuromage 

Resolving patch:
diff --git a/components/gcp/container/component_sdk/python/kfp_component/google/ml_engine/_client.py b/components/gcp/container/component_sdk/python/kfp_component/google/ml_engine/_client.py
--- a/components/gcp/container/component_sdk/python/kfp_component/google/ml_engine/_client.py
+++ b/components/gcp/container/component_sdk/python/kfp_component/google/ml_engine/_client.py
@@ -12,6 +12,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+from functools import wraps
 import logging
 import time
 
@@ -19,12 +20,53 @@
 from googleapiclient import errors
 from ..common import wait_operation_done
 
+
+def _retry(func, tries=5, delay=1):
+    """Retry decorator for methods in MLEngineClient class.
+
+    It bypasses the BrokenPipeError by directly accessing the `_build_client` method
+    and rebuilds `_ml_client` after `delay` seconds.
+
+    Args:
+         tries (int): Total number of retries if BrokenPipeError/IOError is raised.
+         delay (int): Number of seconds to wait between consecutive retries.
+    """
+
+    @wraps(func)
+    def wrapper(self, *args, **kwargs):
+        _tries, _delay = tries, delay
+        while _tries:
+            try:
+                return func(self, *args, **kwargs)
+            except (BrokenPipeError, IOError) as e:
+                _tries -= 1
+                if not _tries:
+                    raise
+
+                logging.warning(
+                    'Caught {}. Retrying in {} seconds...'.format(
+                        e._class__.__name__, _delay
+                    )
+                )
+
+                time.sleep(_delay)
+                # access _build_client method and rebuild Http Client
+                self._build_client()
+
+    return wrapper
+
+
 class MLEngineClient:
     """ Client for calling MLEngine APIs.
     """
+
     def __init__(self):
+        self._build_client()
+
+    def _build_client(self):
         self._ml_client = discovery.build('ml', 'v1', cache_discovery=False)
 
+    @_retry
     def create_job(self, project_id, job):
         """Create a new job.
 
@@ -40,6 +82,7 @@ def create_job(self, project_id, job):
             body = job
         ).execute()
 
+    @_retry
     def cancel_job(self, project_id, job_id):
         """Cancel the specified job.
 
@@ -55,6 +98,7 @@ def cancel_job(self, project_id, job_id):
             },
         ).execute()
 
+    @_retry
     def get_job(self, project_id, job_id):
         """Gets the job by ID.
 
@@ -68,6 +112,7 @@ def get_job(self, project_id, job_id):
         return self._ml_client.projects().jobs().get(
             name=job_name).execute()
 
+    @_retry
     def create_model(self, project_id, model):
         """Creates a new model.
 
@@ -82,6 +127,7 @@ def create_model(self, project_id, model):
             body = model
         ).execute()
 
+    @_retry
     def get_model(self, model_name):
         """Gets a model.
 
@@ -94,6 +140,7 @@ def get_model(self, model_name):
             name = model_name
         ).execute()
 
+    @_retry
     def create_version(self, model_name, version):
         """Creates a new version.
 
@@ -109,6 +156,7 @@ def create_version(self, model_name, version):
             body = version
         ).execute()
 
+    @_retry
     def get_version(self, version_name):
         """Gets a version.
 
@@ -127,6 +175,7 @@ def get_version(self, version_name):
                 return None
             raise
 
+    @_retry
     def delete_version(self, version_name):
         """Deletes a version.
 
@@ -146,11 +195,13 @@ def delete_version(self, version_name):
                 return None
             raise
 
+    @_retry
     def set_default_version(self, version_name):
         return self._ml_client.projects().models().versions().setDefault(
             name = version_name
         ).execute()
 
+    @_retry
     def get_operation(self, operation_name):
         """Gets an operation.
 
@@ -178,6 +229,7 @@ def wait_for_operation_done(self, operation_name, wait_interval):
         return wait_operation_done(
             lambda: self.get_operation(operation_name), wait_interval)
 
+    @_retry
     def cancel_operation(self, operation_name):
         """Cancels an operation.
 


Test patch: