Repository: kubeflow/pipelines

Issue:
VolumeOp doesn't support GC after workflow deletion
The PVC created from VolumeOp is not automatically recycled after workflow is deleted.

I wonder if owner reference can work with PVC when it is set with workflow's reference. If it works, we can provide an option in volumeop to set it when creating the PVC.

If it doesn't work, we should provide a way to either use Argo's volume API or support a delete PVC op in exit handler.
VolumeOp doesn't support GC after workflow deletion
The PVC created from VolumeOp is not automatically recycled after workflow is deleted.

I wonder if owner reference can work with PVC when it is set with workflow's reference. If it works, we can provide an option in volumeop to set it when creating the PVC.

If it doesn't work, we should provide a way to either use Argo's volume API or support a delete PVC op in exit handler.


Discussion:
What is the practical consequence of this? I'm having an issue with bound PVCs from pipelines that ran long ago. 
Hi all,
I'm working on a solution using owner references that seems to be working just fine!
I'll follow up with a PR soon.

/assign elikatsis
As I mentioned in the previous comment, I've been working on solving this issue using `OwnerReferences`.

An OwnerReference requires two basic fields that refer to the actual Kubernetes resource: `name` and `uid`.
Argo enables access to those using [Argo (v2.3.0) variables](https://github.com/argoproj/argo/blob/v2.3.0/docs/variables.md): `{{workflow.name}}`, `{{workflow.uid}}`.

PR kubeflow/pipelines#1720, however, changed `{{workflow.uid}}` so that it points to runID instead, making it impossible to add OwnerReferences.

I'm not sure I understand the rationale behind this. [ cc @IronPan ]
Why don't we use `{{workflow.labels.pipeline/runid}}`?
[ cc @Ark-kun since it's used in the DSL in `RUN_ID_PLACEHOLDER` ]
It feels very odd having `{{workflow.uid}}` modified and not function as it's supposed to (i.e., be replaced with workflow's UUID), especially when there's a valid way to do what we desire.
I would be very interested in this. We have workflows where we sometimes need to dump out fairly large files to disk, but they are only relevant while the component is executing. Currently we have to always manually delete the PVCs afterwards
@Ark-kun @IronPan friendly ping :smile: 
Willing to help, and will send a PR for adding support a delete PVC op.  
@Ark-kun @IronPan 

Hi there! Pinging again for an answer to this comment: https://github.com/kubeflow/pipelines/issues/1779#issuecomment-572001702

Since it's been some time (KFP has had many releases, KF ships with a KFP version containing #1720), can we move to using a different _Argo magic string_ and not override the existing `{{workflow.uid}}`?
Is it OK if I submit a PR fixing that behavior?

If we go with `{{workflow.labels.pipeline/runid}}` rather than a new `{{kfp.run.id}}`, there will also be no problem with incompatibilities between SDK and apiserver.
I generally agree with the suggestion.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

I don't think it's fixed now, please keep open until we fix.
Is there any workaround to remove the actual volume (disk) after the pipeline finishes? 
Using
```python
my_volume = dsl.VolumeOp(
        name="create-volume",
        resource_name="asd",
        size="10Gi",
        modes=dsl.VOLUME_MODE_RWO,
    )
dsl.ResourceOp(
        name="delete-volume",
        k8s_resource=my_volume.k8s_resource,
        action="delete"
    )
```

still leaves the unused disk in GCE.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

This is still a sorely-needed feature for us, would be good to keep the ticket open until it is resolved.
Agree with @jackwhelpton 
I would reopen it. 
Thank you
Hey, did anyone found any workaround for this?

@kubeflow/wg-pipeline-leads I think this is a well-known issue and all the cloud-provider should get the same issue.

As I can see, we met similar issue here. Can you put some efforts on this issue and I can help as well?

/cc @Bobgy 
/assign @chensun 
Can you take a look?
Looks like this is requested by a lot of people
What is the practical consequence of this? I'm having an issue with bound PVCs from pipelines that ran long ago. 
Hi all,
I'm working on a solution using owner references that seems to be working just fine!
I'll follow up with a PR soon.

/assign elikatsis
As I mentioned in the previous comment, I've been working on solving this issue using `OwnerReferences`.

An OwnerReference requires two basic fields that refer to the actual Kubernetes resource: `name` and `uid`.
Argo enables access to those using [Argo (v2.3.0) variables](https://github.com/argoproj/argo/blob/v2.3.0/docs/variables.md): `{{workflow.name}}`, `{{workflow.uid}}`.

PR kubeflow/pipelines#1720, however, changed `{{workflow.uid}}` so that it points to runID instead, making it impossible to add OwnerReferences.

I'm not sure I understand the rationale behind this. [ cc @IronPan ]
Why don't we use `{{workflow.labels.pipeline/runid}}`?
[ cc @Ark-kun since it's used in the DSL in `RUN_ID_PLACEHOLDER` ]
It feels very odd having `{{workflow.uid}}` modified and not function as it's supposed to (i.e., be replaced with workflow's UUID), especially when there's a valid way to do what we desire.
I would be very interested in this. We have workflows where we sometimes need to dump out fairly large files to disk, but they are only relevant while the component is executing. Currently we have to always manually delete the PVCs afterwards
@Ark-kun @IronPan friendly ping :smile: 
Willing to help, and will send a PR for adding support a delete PVC op.  
@Ark-kun @IronPan 

Hi there! Pinging again for an answer to this comment: https://github.com/kubeflow/pipelines/issues/1779#issuecomment-572001702

Since it's been some time (KFP has had many releases, KF ships with a KFP version containing #1720), can we move to using a different _Argo magic string_ and not override the existing `{{workflow.uid}}`?
Is it OK if I submit a PR fixing that behavior?

If we go with `{{workflow.labels.pipeline/runid}}` rather than a new `{{kfp.run.id}}`, there will also be no problem with incompatibilities between SDK and apiserver.
I generally agree with the suggestion.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

I don't think it's fixed now, please keep open until we fix.
Is there any workaround to remove the actual volume (disk) after the pipeline finishes? 
Using
```python
my_volume = dsl.VolumeOp(
        name="create-volume",
        resource_name="asd",
        size="10Gi",
        modes=dsl.VOLUME_MODE_RWO,
    )
dsl.ResourceOp(
        name="delete-volume",
        k8s_resource=my_volume.k8s_resource,
        action="delete"
    )
```

still leaves the unused disk in GCE.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

This is still a sorely-needed feature for us, would be good to keep the ticket open until it is resolved.
Agree with @jackwhelpton 
I would reopen it. 
Thank you
Hey, did anyone found any workaround for this?

@kubeflow/wg-pipeline-leads I think this is a well-known issue and all the cloud-provider should get the same issue.

As I can see, we met similar issue here. Can you put some efforts on this issue and I can help as well?

/cc @Bobgy 
/assign @chensun 
Can you take a look?
Looks like this is requested by a lot of people

Resolving patch:
diff --git a/sdk/python/kfp/dsl/_resource_op.py b/sdk/python/kfp/dsl/_resource_op.py
--- a/sdk/python/kfp/dsl/_resource_op.py
+++ b/sdk/python/kfp/dsl/_resource_op.py
@@ -31,6 +31,7 @@ class Resource(object):
       "merge_strategy": "str",
       "success_condition": "str",
       "failure_condition": "str",
+      "set_owner_reference": "bool",
       "manifest": "str",
       "flags": "list[str]"
   }
@@ -39,6 +40,7 @@ class Resource(object):
       "merge_strategy": "str",
       "success_condition": "str",
       "failure_condition": "str",
+      "set_owner_reference": "bool",
       "manifest": "str",
       "flags": "list[str]"
   }
@@ -47,6 +49,7 @@ class Resource(object):
       "merge_strategy": "mergeStrategy",
       "success_condition": "successCondition",
       "failure_condition": "failureCondition",
+      "set_owner_reference": "setOwnerReference",
       "manifest": "manifest",
       "flags": "flags"
   }
@@ -56,6 +59,7 @@ def __init__(self,
                merge_strategy: str = None,
                success_condition: str = None,
                failure_condition: str = None,
+               set_owner_reference: bool = None,
                manifest: str = None,
                flags: Optional[List[str]] = None):
     """Create a new instance of Resource"""
@@ -63,6 +67,7 @@ def __init__(self,
     self.merge_strategy = merge_strategy
     self.success_condition = success_condition
     self.failure_condition = failure_condition
+    self.set_owner_reference = set_owner_reference
     self.manifest = manifest
     self.flags = flags
 
@@ -98,6 +103,7 @@ def __init__(self,
                merge_strategy: str = None,
                success_condition: str = None,
                failure_condition: str = None,
+               set_owner_reference: bool = None,
                attribute_outputs: Optional[Dict[str, str]] = None,
                flags: Optional[List[str]] = None,
                **kwargs):
@@ -135,6 +141,7 @@ def __init__(self,
         "merge_strategy": merge_strategy,
         "success_condition": success_condition,
         "failure_condition": failure_condition,
+        "set_owner_reference": set_owner_reference,
         "flags": flags
     }
     # `resource` prop in `io.argoproj.workflow.v1alpha1.Template`


Test patch:
diff --git a/sdk/python/tests/compiler/compiler_tests.py b/sdk/python/tests/compiler/compiler_tests.py
--- a/sdk/python/tests/compiler/compiler_tests.py
+++ b/sdk/python/tests/compiler/compiler_tests.py
@@ -72,7 +72,8 @@ def my_pipeline(msg1, json, kind, msg2='value2'):
             name="resource"
           )
         ),
-        attribute_outputs={"out": json}
+        attribute_outputs={"out": json},
+        set_owner_reference=True
       )
       golden_output = {
         'container': {
@@ -147,7 +148,8 @@ def my_pipeline(msg1, json, kind, msg2='value2'):
             "kind: '{{inputs.parameters.kind}}'\n"
             "metadata:\n"
             "  name: resource\n"
-          )
+          ),
+          'setOwnerReference': True
         }
       }