Repository: kubeflow/pipelines

Issue:
mount_pvc failing when pipeline includes ResourceOps
### What steps did you take:

Created a persistent volume claim using a `VolumeOp`:

```python
    vop = dsl.VolumeOp(
        name="shared-data",
        resource_name="shared-data",
        size="100G",
        storage_class="nfs",
    ).set_display_name("Prepare volume")
```

Attempted to mount the PVC to every operation in a pipeline using:

```python
    dsl.get_pipeline_conf().add_op_transformer(
        mount_pvc(pvc_name=vop.volume.persistent_volume_claim.claim_name, volume_mount_path=MOUNT_PATH)
    )
```

### What happened:
During compilation, the `add_op_transformer` attempts to invoke the `add_volume_mount` method on the `VolumeOp`, producing an error:

```
AttributeError: 'VolumeOp' object has no attribute 'add_volume_mount'
```

### What did you expect to happen:

The documentation states that transfomers take and return a `ContainerOp`:

https://kubeflow-pipelines.readthedocs.io/en/latest/source/kfp.dsl.html#kfp.dsl.PipelineConf.add_op_transformer

Given that, I would expect them to be bypassed for `VolumeOp` (or other `ResourceOp` operations), and hence this compilation should succeed.

### Environment:
<!-- Please fill in those that seem relevant. -->

KFP version: Build commit: 9c16e12

KFP SDK version: <!-- Please attach the output of this shell command: $pip list | grep kfp -->
kfp                      0.5.1
kfp-server-api           0.3.0

### Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

/kind bug
<!-- Please include labels by uncommenting them to help us better triage issues, choose from the following -->
<!--
// /area frontend
// /area backend
// /area sdk
// /area testing
// /area engprod
-->



Discussion:
Hi @jackwhelpton,

The documentation seems wrong there. Since there is commonality between the ops, it's the op transformer's job to check before applying something exclusive to a specific kind of ops.

Could you write a few-lines-wrapper around `mount_pvc` to do that check?
It should be something like
```python
def mount_pvc_transformer(pvc_name, mount_path):
    def transformer(op):
        if isinstance(op, dsl.ContainerOp):
            mount_pvc(pvc_name=pvc_name, volume_mount_path=mount_path)
    return transformer
```

Also, instead of `mount_pvc` you could use the following, since `mount_pvc` can fail if you attempt to mount multiple volumes with it and you don't have advanced K8s knowledge.
```python
op.add_pvolumes({mount_path: dsl.PipelineVolume(pvc=pvc_name)})
```

Resolving patch:
diff --git a/sdk/python/kfp/dsl/_pipeline.py b/sdk/python/kfp/dsl/_pipeline.py
--- a/sdk/python/kfp/dsl/_pipeline.py
+++ b/sdk/python/kfp/dsl/_pipeline.py
@@ -102,9 +102,9 @@ def set_ttl_seconds_after_finished(self, seconds: int):
     """
     self.ttl_seconds_after_finished = seconds
     return self
-  
-  def set_default_pod_node_selector(self, label_name: str, value: str): 
-    """Add a constraint for nodeSelector for a pipeline. Each constraint is a key-value pair label. For the 
+
+  def set_default_pod_node_selector(self, label_name: str, value: str):
+    """Add a constraint for nodeSelector for a pipeline. Each constraint is a key-value pair label. For the
       container to be eligible to run on a node, the node must have each of the constraints appeared
       as labels.
 
@@ -114,7 +114,7 @@ def set_default_pod_node_selector(self, label_name: str, value: str):
     """
     self.default_pod_node_selector[label_name] = value
     return self
-  
+
 
   def set_image_pull_policy(self, policy: str):
     """Configures the default image pull policy
@@ -128,9 +128,10 @@ def set_image_pull_policy(self, policy: str):
 
   def add_op_transformer(self, transformer):
     """Configures the op_transformers which will be applied to all ops in the pipeline.
+    The ops can be ResourceOp, VolumenOp, or ContainerOp.
 
     Args:
-      transformer: a function that takes a ContainOp as input and returns a ContainerOp
+      transformer: a function that takes a kfp Op as input and returns a kfp Op
     """
     self.op_transformers.append(transformer)
 
diff --git a/sdk/python/kfp/onprem.py b/sdk/python/kfp/onprem.py
--- a/sdk/python/kfp/onprem.py
+++ b/sdk/python/kfp/onprem.py
@@ -1,7 +1,7 @@
 
 def mount_pvc(pvc_name='pipeline-claim', volume_name='pipeline', volume_mount_path='/mnt/pipeline'):
     """
-        Modifier function to apply to a Container Op to simplify volume, volume mount addition and 
+        Modifier function to apply to a Container Op to simplify volume, volume mount addition and
         enable better reuse of volumes, volume claims across container ops.
         Usage:
             train = train_op(...)
@@ -9,6 +9,10 @@ def mount_pvc(pvc_name='pipeline-claim', volume_name='pipeline', volume_mount_pa
     """
     def _mount_pvc(task):
         from kubernetes import client as k8s_client
+        # there can be other ops in a pipeline (e.g. ResourceOp, VolumeOp)
+        # refer to #3906
+        if not hasattr(task, "add_volume") or not hasattr(task, "add_volume_mount"):
+            return task
         local_pvc = k8s_client.V1PersistentVolumeClaimVolumeSource(claim_name=pvc_name)
         return (
             task


Test patch: