Repository: kubeflow/pipelines

Issue:
Cannot use resource request based on PipelineParam
We are trying to build a generic pipeline that will have some steps with configurable resources. This is needed because based on what is being run it might need more or less cpu, memory and etc.

Consider the pipeline described bellow:

```python
@dsl.pipeline(
    name='Generic Pipe'
)
def pipeline_definition(memory_request: str = '8G', cpu_request: str = '4'):
    op = ContainerOp(...)

    op.set_memory_request(memory_request)\
      .set_cpu_request(cpu_request)
```

This pipeline will fail to match the expected resource Regex and would not be successfully uploaded. This is what would be generated on the yaml file:

```yaml
 resources:
    requests:
        cpu: '{{inputs.parameters.cpu-request}}'
        memory: '{{inputs.parameters.memory-request}}'
```

And here is the error:

```text
kfp_server_api.rest.ApiException: (500)
Reason: Internal Server Error
HTTP response headers: HTTPHeaderDict({'Server': 'nginx/1.15.6', 'Date': 'Mon, 26 Aug 2019 21:46:03 GMT', 'Content-Type': 'text/plain; charset=utf-8', 'Content-Length': '601', 'Connection': 'keep-alive', 'x-powered-by': 'Express', 'x-envoy-upstream-service-time': '55'})
HTTP response body: {"error_message":"Error creating pipeline: Create pipeline failed: Failed to get parameters from the workflow: InvalidInputError: Failed to parse the parameter.: error unmarshaling JSON: while decoding JSON: quantities must match the regular expression '^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$'","error_details":"Error creating pipeline: Create pipeline failed: Failed to get parameters from the workflow: InvalidInputError: Failed to parse the parameter.: error unmarshaling JSON: while decoding JSON: quantities must match the regular expression '^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$'"}
```




Discussion:
@IronPan Do you have any idea of how to solve this? 
Hello,

This issue is related to argoproj/argo#703 and it is not fixed yet.
It looks like https://github.com/argoproj/argo/pull/1687 might fix this. I'm very new to kubeflow - can anyone explain the workaround in https://github.com/argoproj/argo/issues/703 - is it available to kubeflow-land? 
@Ark-kun, any suggested workaround?
This is a problem with trying to use templates in structures with strict schemas. @numerology had the same issue when trying to template protobufs.

The addition pf PatchPodSpec https://github.com/argoproj/argo/pull/1687 in Argo v2.4.2 makes it possible to fix this, although this will require some amount of work on the SDK side, upgrade to Argo v2.4.2. It will also be backwards incompatible with existing KFP backends (since older versions of Argo do not support the feature).
I'm getting blocked by this too. We're also trying to configure GPU dynamically, but that's running into problems because `add_node_selector_constraint` can't take PipelineParams: I was hoping I could use a previous parse step to build the node selector and then apply it using `parse_task.outputs`.

Would that be covered by this ticket, or need a new one, or is there a workaround I'm missing?
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

This issue has been automatically closed because it has not had recent activity. Please comment "/reopen" to reopen it.

/reopen

This would still be a useful feature to have.
@jackwhelpton: You can't reopen an issue/PR unless you authored it or you are a collaborator.

<details>

In response to [this](https://github.com/kubeflow/pipelines/issues/1956#issuecomment-663157124):

>/reopen
>
>This would still be a useful feature to have.


Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
</details>
/reopen
for @jackwhelpton 
@Bobgy: Reopened this issue.

<details>

In response to [this](https://github.com/kubeflow/pipelines/issues/1956#issuecomment-663321667):

>/reopen
>for @jackwhelpton 


Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
</details>
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

Argo was upgraded to 2.7.5 in https://github.com/kubeflow/pipelines/pull/3537 so now Argo's pod spec patching (https://github.com/argoproj/argo/pull/1687) is available to implement a fix in the Pipelines SDK.
/assign @chensun 
FYI, this is the top open issue with the most thumb ups
@chensun 

I don't think this will be super complicated to fix. I guess it's currently blocked by this allowlist [here](https://github.com/kubeflow/pipelines/blob/f7874d38ffe1d6eaf13ddffef5c61c44217b47cf/sdk/python/kfp/dsl/_container_op.py#L1031). We can take a look when we have some cycle.
/assign
I will take a look at this. 
Is https://github.com/kubeflow/pipelines/pull/5239 a better alternative now?
> Is https://github.com/kubeflow/pipelines/pull/5239 a better alternative now?

I don't fully follow how the additions of K8s create/apply components would allow the user to specify the resource specs as input parameters to a pipeline? Could you elaborate @Bobgy? 
Sorry, I misread the issue's description. Please ignore my comment above

Resolving patch:
diff --git a/samples/core/resource_spec/runtime_resource_request.py b/samples/core/resource_spec/runtime_resource_request.py
new file mode 100644
--- /dev/null
+++ b/samples/core/resource_spec/runtime_resource_request.py
@@ -0,0 +1,48 @@
+# Copyright 2021 The Kubeflow Authors
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import kfp
+from kfp import dsl, components
+from typing import NamedTuple
+
+@components.create_component_from_func
+def training_op(n: int) -> int:
+    # quickly allocate a lot of memory to verify memory is enough
+    a = [i for i in range(n)]
+    return len(a)
+
+@components.create_component_from_func
+def generate_resouce_request() -> NamedTuple('output', [('memory', str), ('cpu', str)]):
+    '''Returns the memory and cpu request'''
+    from collections import namedtuple
+    
+    resouce_output = namedtuple('output', ['memory', 'cpu'])
+    return resouce_output('500Mi', '200m')
+
+@dsl.pipeline(
+    name='Runtime resource request pipeline',
+    description='An example on how to make resource requests at runtime.'
+)
+def resource_request_pipeline(n: int = 11234567):
+    resouce_task = generate_resouce_request()
+    traning_task = training_op(n)\
+        .set_memory_limit(resouce_task.outputs['memory'])\
+        .set_cpu_limit(resouce_task.outputs['cpu'])\
+        .set_cpu_request('200m')
+
+    # Disable cache for KFP v1 mode.
+    traning_task.execution_options.caching_strategy.max_cache_staleness = 'P0D'
+
+if __name__ == '__main__':
+    kfp.compiler.Compiler().compile(resource_request_pipeline, __file__ + '.yaml')
\ No newline at end of file
diff --git a/sdk/python/kfp/compiler/_op_to_template.py b/sdk/python/kfp/compiler/_op_to_template.py
--- a/sdk/python/kfp/compiler/_op_to_template.py
+++ b/sdk/python/kfp/compiler/_op_to_template.py
@@ -288,6 +288,24 @@ def _op_to_template(op: BaseOp):
         template['volumes'] = [convert_k8s_obj_to_json(volume) for volume in processed_op.volumes]
         template['volumes'].sort(key=lambda x: x['name'])
 
+    # Runtime resource requests
+    if isinstance(op, dsl.ContainerOp) and ('resources' in op.container.keys()):
+        podSpecPatch = {}
+        for setting, val in op.container['resources'].items():
+            for resource, param in val.items():
+                if (resource in ['cpu', 'memory']) and re.match('^{{inputs.parameters.*}}$', param):
+                    if not 'containers' in podSpecPatch:
+                        podSpecPatch = {'containers':[{'name':'main', 'resources':{}}]}
+                    if setting not in podSpecPatch['containers'][0]['resources']:
+                        podSpecPatch['containers'][0]['resources'][setting] = {resource: param}
+                    else:
+                        podSpecPatch['containers'][0]['resources'][setting][resource] = param
+                    del template['container']['resources'][setting][resource]
+                    if not template['container']['resources'][setting]:
+                        del template['container']['resources'][setting]
+        if podSpecPatch:
+            template['podSpecPatch'] = json.dumps(podSpecPatch)
+            
     if isinstance(op, dsl.ContainerOp) and op._metadata and not op.is_v2:
         template.setdefault('metadata', {}).setdefault('annotations', {})['pipelines.kubeflow.org/component_spec'] = json.dumps(op._metadata.to_dict(), sort_keys=True)
 
diff --git a/sdk/python/kfp/dsl/_container_op.py b/sdk/python/kfp/dsl/_container_op.py
--- a/sdk/python/kfp/dsl/_container_op.py
+++ b/sdk/python/kfp/dsl/_container_op.py
@@ -288,27 +288,29 @@ def add_resource_request(self, resource_name, value) -> 'Container':
     self.resources.requests.update({resource_name: value})
     return self
 
-  def set_memory_request(self, memory) -> 'Container':
+  def set_memory_request(self, memory: Union[str,  _pipeline_param.PipelineParam]) -> 'Container':
     """Set memory request (minimum) for this operator.
 
     Args:
-      memory: a string which can be a number or a number followed by one of
+      memory(Union[str, PipelineParam]): a string which can be a number or a number followed by one of
         "E", "P", "T", "G", "M", "K".
     """
 
-    self._validate_size_string(memory)
+    if not isinstance(memory,_pipeline_param.PipelineParam):  
+      self._validate_size_string(memory)
     return self.add_resource_request('memory', memory)
 
-  def set_memory_limit(self, memory) -> 'Container':
+  def set_memory_limit(self, memory: Union[str,  _pipeline_param.PipelineParam]) -> 'Container':
     """Set memory limit (maximum) for this operator.
 
     Args:
-      memory: a string which can be a number or a number followed by one of
+      memory(Union[str, PipelineParam]): a string which can be a number or a number followed by one of
         "E", "P", "T", "G", "M", "K".
     """
-    self._validate_size_string(memory)
-    if self._container_spec:
-      self._container_spec.resources.memory_limit = _get_resource_number(memory)
+    if not isinstance(memory,_pipeline_param.PipelineParam):
+      self._validate_size_string(memory)
+      if self._container_spec:
+        self._container_spec.resources.memory_limit = _get_resource_number(memory)
     return self.add_resource_limit('memory', memory)
 
   def set_ephemeral_storage_request(self, size) -> 'Container':
@@ -331,27 +333,29 @@ def set_ephemeral_storage_limit(self, size) -> 'Container':
     self._validate_size_string(size)
     return self.add_resource_limit('ephemeral-storage', size)
 
-  def set_cpu_request(self, cpu) -> 'Container':
+  def set_cpu_request(self, cpu: Union[str,  _pipeline_param.PipelineParam]) -> 'Container':
     """Set cpu request (minimum) for this operator.
 
     Args:
-      cpu: A string which can be a number or a number followed by "m", which
+      cpu(Union[str, PipelineParam]): A string which can be a number or a number followed by "m", which
         means 1/1000.
     """
-
-    self._validate_cpu_string(cpu)
+    if not isinstance(cpu,_pipeline_param.PipelineParam):  
+      self._validate_cpu_string(cpu)
     return self.add_resource_request('cpu', cpu)
 
-  def set_cpu_limit(self, cpu) -> 'Container':
+  def set_cpu_limit(self, cpu: Union[str,  _pipeline_param.PipelineParam]) -> 'Container':
     """Set cpu limit (maximum) for this operator.
 
     Args:
-      cpu: A string which can be a number or a number followed by "m", which
+      cpu(Union[str, PipelineParam]): A string which can be a number or a number followed by "m", which
         means 1/1000.
     """
-    self._validate_cpu_string(cpu)
-    if self._container_spec:
-      self._container_spec.resources.cpu_limit = _get_cpu_number(cpu)
+
+    if not isinstance(cpu,_pipeline_param.PipelineParam):  
+      self._validate_cpu_string(cpu)
+      if self._container_spec:
+        self._container_spec.resources.cpu_limit = _get_cpu_number(cpu)
     return self.add_resource_limit('cpu', cpu)
 
   def set_gpu_limit(self, gpu, vendor='nvidia') -> 'Container':


Test patch:
diff --git a/samples/core/resource_spec/runtime_resource_request_test.py b/samples/core/resource_spec/runtime_resource_request_test.py
new file mode 100644
--- /dev/null
+++ b/samples/core/resource_spec/runtime_resource_request_test.py
@@ -0,0 +1,48 @@
+# Copyright 2021 The Kubeflow Authors
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import kfp
+from .runtime_resource_request import resource_request_pipeline
+from ...test.util import run_pipeline_func, TestCase
+
+
+def EXPECTED_OOM(run_id, run, **kwargs):
+    '''confirms a sample test case is failing, because of OOM '''
+    assert run.status == 'Failed'
+
+
+run_pipeline_func([
+    TestCase(
+        pipeline_func=resource_request_pipeline,
+        mode=kfp.dsl.PipelineExecutionMode.V1_LEGACY,
+    ),
+    # TODO: blocked by https://github.com/kubeflow/pipelines/issues/5835
+    # TestCase(
+    #     pipeline_func=resource_request_pipeline,
+    #     mode=kfp.dsl.PipelineExecutionMode.V2_COMPATIBLE,
+    # ),
+    TestCase(
+        pipeline_func=resource_request_pipeline,
+        mode=kfp.dsl.PipelineExecutionMode.V1_LEGACY,
+        arguments={'n': 21234567},
+        verify_func=EXPECTED_OOM,
+    ),
+    # TODO: blocked by https://github.com/kubeflow/pipelines/issues/5835
+    # TestCase(
+    #     pipeline_func=resource_request_pipeline,
+    #     mode=kfp.dsl.PipelineExecutionMode.V2_COMPATIBLE,
+    #     arguments={'n': 21234567},
+    #     verify_func=EXPECTED_OOM,
+    # ),
+])
diff --git a/samples/test/config.yaml b/samples/test/config.yaml
--- a/samples/test/config.yaml
+++ b/samples/test/config.yaml
@@ -40,6 +40,8 @@
   path: samples.core.loop_parallelism.loop_parallelism_test
 - name: resource_spec
   path: samples.core.resource_spec.resource_spec_test
+- name: runtime_resource_spec
+  path: samples.core.resource_spec.runtime_resource_request_test
 - name: xgboost_sample
   path: samples.core.XGBoost.xgboost_sample_test
 
diff --git a/sdk/python/tests/compiler/compiler_tests.py b/sdk/python/tests/compiler/compiler_tests.py
--- a/sdk/python/tests/compiler/compiler_tests.py
+++ b/sdk/python/tests/compiler/compiler_tests.py
@@ -406,6 +406,20 @@ def my_pipeline():
       self.assertEqual(template['retryStrategy']['backoff']['maxDuration'], backoff_max_duration)
 
 
+  def test_py_runtime_memory_request(self):
+      """Test memory request."""
+
+      def my_pipeline(memory: str, cpu: str):
+        some_op().set_cpu_request(memory)
+
+      workflow = kfp.compiler.Compiler()._create_workflow(my_pipeline)
+      name_to_template = {template['name']: template for template in workflow['spec']['templates']}
+      main_dag_tasks = name_to_template[workflow['spec']['entrypoint']]['dag']['tasks']
+      template = name_to_template[main_dag_tasks[0]['template']]
+
+      self.assertEqual(template['podSpecPatch'], '{"containers": [{"name": "main", "resources": {"requests": {"cpu": "{{inputs.parameters.memory}}"}}}]}')
+      
+
   def test_py_retry_policy_invalid(self):
       def my_pipeline():
           some_op().set_retry(2, 'Invalid')