Repository: kestra-io/kestra
Issue: Add a new Runnable task — DynamicTask allowing to template the entire task definition even for non-dynamic properties
Labels: enhancement
customer-request

Sometimes, we can't make specific properties dynamic because we want to offer type validation for integers, floats, booleans, or complex objects. This issue keeps coming more and more across various user requests (Databricks, Kubernetes PodCreate, and more).

The challenge is that: 
- _some_ users want to make almost every property dynamic. They care about making all inputs dynamic more than type safety. 
- for another type of user, that type safety and syntax/type validation while typing the declarative flow definition are important.


To offer a balanced approach, we may offer a new core task that allows users to fully dynamically generate a task definition. 


Name TBD, possible options:
- TemplatedTask
- DynamicTask

Given we disable [recursive rendering](https://kestra.io/docs/migrations/recursive-rendering#how-to-keep-the-previous-behavior) in the app globally, here it's also set to false by default.

```yaml
id: template
namespace: dev

inputs:
  - name: restartPolicy
    type: STRING

  - name: taskId
    type: STRING

tasks:
  - id: dynamic
    type: io.kestra.core.tasks.templating.TemplatedTask
    spec: |
      id: "{{ inputs.taskId }}"
      type: io.kestra.plugin.kubernetes.PodCreate
      namespace: default
      metadata:
        labels:
          my-label: my-value
      spec:
        containers:
        - name: unittest
          image: debian:stable-slim
          command: 
            - 'bash' 
            - '-c'
            - 'for i in {1..10}; do echo $i; sleep 0.1; done'
          {% if inputs.restartPolicy}
          restartPolicy: "{{ inputs.restartPolicy }}"
          {% endif %}
```

Note how, in this example, even task ID or task type can be dynamically generated at runtime based on user inputs, environment variables and more. 

In this example, the flow topology view will show the `DynamicTask` as a task type, while the execution topology will dynamically render that template into the right task type. 

Issue comments:
This would enhance the use of templated subflows, allowing to create subflows that use inputs to define it's behaviour.
TBD consider making it just one task as the implementation would be much easier/faster
Here's an example that could kind of work out of the box, dependent on the deserialization of YAML and type verification.

```yaml
id: nested-parameters-from-inputs
namespace: test

inputs:
  - name: imagePullSecrets
    type: STRING
    # defaults: ""
    description: "Kubernetes GitLab Container Registry API access token"
    required: true

  # the following represents the object resources with nested structure as supported by Kestra
  - name: resources.limits.cpu
    type: STRING
    defaults: "1"
    required: true

  - name: resources.limits.memory
    type: STRING
    defaults: "1000Mi"
    required: true

  - name: resources.requests.cpu
    type: STRING
    required: false # note that this is optional and when not filled, it should be skipped in the definition (could be with a filter before passing the argument)

  - name: resources.requests.memory
    type: STRING
    required: false # note that this is optional and when not filled, it should be skipped in the definition (could be with a filter before passing the argument)

# using variables here to support other data types not supported by inputs, specifically lists
variables:
  env: # interpreted as an array of objects
    - name: VARIABLE_1
      value: "VALUE_1"
    - name: VARIABLE_2
      value: "VALUE_2"

  command: # interpreted as an array of strings
    - 'bash' 
    - '-c'
    - 'for i in {1..10}; do echo $i; sleep 0.1; done'

tasks:
  - id: pod
    type: "io.kestra.plugin.kubernetes.PodCreate"
    namespace: kestra
    spec:
      containers:
      - name: unittest
        image: debian:stable-slim

        # k8s command by definition is a list of strings
        # here we should be able to expand variables.command, which is also a list of strings
        command: {{ vars.command }}

        # k8s env by definition is a list of objects (name, value as keys)
        # here we should be able to expand variables.env, which is also a list of objects
        env: {{ vars.env }} 

        # k8s resources is a complex nested object
        # since in the input variable "resources" we have also defined a nested input parameter
        # that follows the structure, we should be able to deserialize it just like YAML definition
        resources: {{ inputs.resources }}
        # here we should perhaps filter out any attributes with null values, when they are not defined in the inputs.

        # example of the original idea in this issue
        # optional, string based deserialization, maybe not the best way to handle this
        # this would require deserialization of a string to a nested YAML format
        imagePullSecrets: | 
          - name: "{{ inputs.imagePullSecrets }}"

```

All except for the last example should actually be similar to Kestra's best practice, however currently, this way it does not really work.

Even though the nested input and variables are correctly loaded as JSON objects/arrays, then the type verification and deserialization process happens in an order that prohibits this type of use.

The last option which is an example from Anna's post is one workaround to implement the end goal, but I'd actually prefer a more straight forward approach - not having to hard-code all the properties of the task definition, but pass them along as (almost) correctly typed objects.

This would allow manipulation of the objects between inputs, passing them to subflows and dynamically altering where necessary.
Perhaps a more concrete example could be the following, closely resembling our use-case.

### Templated subflow - a common flow to be called from specific implementations

```yaml
id: templated-podcreate
namespace: test

inputs:
  - name: env
    type: JSON
    default: "[]"
    required: false
    description: "Optionally pass additional environment variables to PodCreate"

  - name: command
    type: JSON
    default: "[]"
    required: true
    description: "Pass specific commands to PodCreate"

  - name: imagePullSecrets
    type: STRING
    # defaults: ""
    required: true

  # the following represents the object resources with nested structure as supported by Kestra
  - name: resources.limits.cpu
    type: STRING
    defaults: "1"
    required: true

  - name: resources.limits.memory
    type: STRING
    defaults: "1000Mi"
    required: true

  - name: resources.requests.cpu
    type: STRING
    required: false # note that this is optional and when not filled, it should be skipped in the definition (could be with a filter before passing the argument)

  - name: resources.requests.memory
    type: STRING
    required: false # note that this is optional and when not filled, it should be skipped in the definition (could be with a filter before passing the argument)

variables:
  common_env: # here we define common environment variables that are added to the input env's
    - name: COMMON_1
      value: VALUE


tasks:
  - id: pod
    type: "io.kestra.plugin.kubernetes.PodCreate"
    namespace: kestra
    spec:
      containers:
      - name: unittest
        image: debian:stable-slim

        # k8s command by definition is a list of strings
        # here we should be able to expand variables.command, which is also a list of strings
        command: {{ inputs.command }}

        # k8s env by definition is a list of objects (name, value as keys)
        # here we should be able to expand variables.common_env and add inputs.env, which is also a list of objects
        env: {{ vars.common_env | merge(inputs.env) }}

        # k8s resources is a complex nested object
        # since in the input variable "resources" we have also defined a nested input parameter
        # that follows the structure, we should be able to deserialize it just like YAML definition
        resources: {{ inputs.resources }}
        # here we should perhaps filter out any attributes with null values, when they are not defined in the inputs.

        # example of the original idea in this issue
        # optional, string based deserialization, maybe not the best way to handle this
        # this would require deserialization of a string to a nested YAML format
        # this could be done similarily as with env's example above
        imagePullSecrets: |
          - name: "common-secret" # predefined common pullsecret
          - name: "{{ inputs.imagePullSecrets }}" # string here

```

### Specific flow that itself calls the templated subflow

```yaml
id: run-podcreate
namespace: test

variables:
  env: # interpreted as an array of objects
    - name: VARIABLE_1
      value: "VALUE_1"
    - name: VARIABLE_2
      value: "VALUE_2"

  command: # interpreted as an array of strings
    - 'bash'
    - '-c'
    - 'for i in {1..10}; do echo $i; sleep 0.1; done'

tasks:
  - id: "load-table-kubernetes"
    type: "io.kestra.core.tasks.flows.Subflow"
    flowId: templated-podcreate
    namespace: test
    inputs: 
      env: {{ vars.env }} # passing additional envs to subflow that will be merged with common
      command: {{ vars.command }} # our specific command
      imagePullSecrets: "mypullsecret" # our specific pullsecret

      # resources.limits.cpu:       "1" # not passing, using default defined in subflow
      resources.limits.memory:    "128" # overwriting default defined in subflow
      resources.requests.cpu:     "0.5" # assigning a value to subflow optional input
      # resources.requests.memory:  ""  # not passing, not defined in subflow either, thus should not be passed to k8s
```

### Variable deserialization
Now as for variable interpolation or parsing goes, no preferences here, instead of:
```
        resources: {{ inputs.resources }}
```
it would be fine to use a filter
```
        resources: {{ inputs.resources | objectify }}
```
or a function, such as:
```
        resources: {{ json(inputs.resources) }}
```
or something like:
```
        resources: {{ deserialize(inputs.resources) }}
```

Thus type safety should remain, but it would be possible to pass complex parameters as values to task attributes like it is now possible with strings.
hi @kriko, thanks for all the suggestions. We plan to add the `DynamicTask` as it will allow maximum flexibility for more advanced use cases like this one. You could use it short term until we introduce a proper KUBERNETES runner that will allow a much nicer way to specify resource definition for a pod - WDYT? 

The KUBERNETES runner will not suffer from these issues, everything will be typed properly. 

Pull request review comments:
```suggestion
    title = "This task's `spec` property allows you to fully template all task properties using Kestra's Pebble templating. This way, all task properties and their values can be dynamically rendered based on your custom inputs, variables, and outputs from other tasks!"
```

Patch that resolved the issue:
diff --git a/core/src/main/java/io/kestra/core/tasks/templating/TemplatedTask.java b/core/src/main/java/io/kestra/core/tasks/templating/TemplatedTask.java
new file mode 100644
index 0000000000..2a6398f26b
--- /dev/null
+++ b/core/src/main/java/io/kestra/core/tasks/templating/TemplatedTask.java
@@ -0,0 +1,64 @@
+package io.kestra.core.tasks.templating;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import io.kestra.core.exceptions.IllegalVariableEvaluationException;
+import io.kestra.core.models.annotations.Example;
+import io.kestra.core.models.annotations.Plugin;
+import io.kestra.core.models.annotations.PluginProperty;
+import io.kestra.core.models.tasks.*;
+import io.kestra.core.runners.RunContext;
+import io.kestra.core.serializers.JacksonMapper;
+import io.swagger.v3.oas.annotations.media.Schema;
+import jakarta.validation.constraints.NotNull;
+import lombok.EqualsAndHashCode;
+import lombok.Getter;
+import lombok.NoArgsConstructor;
+import lombok.ToString;
+import lombok.experimental.SuperBuilder;
+
+@SuperBuilder
+@ToString
+@EqualsAndHashCode
+@Getter
+@NoArgsConstructor
+@Schema(
+    title = "This task's `spec` property allows you to fully template all task properties using Kestra's Pebble templating. This way, all task properties and their values can be dynamically rendered based on your custom inputs, variables, and outputs from other tasks!"
+)
+@Plugin(
+    examples = {
+        @Example(
+            code = {
+                """
+                    spec: |
+                      type: io.kestra.plugin.fs.http.Download
+                      {{ task.property }}: {{ task.value }}"""
+            }
+        )
+    }
+)
+public class TemplatedTask extends Task implements RunnableTask<Output> {
+    private static final ObjectMapper OBJECT_MAPPER = JacksonMapper.ofYaml();
+
+    @NotNull
+    @PluginProperty(dynamic = true)
+    @Schema(title = "The templated task specification")
+    private String spec;
+
+    @Override
+    public Output run(RunContext runContext) throws Exception {
+        String taskSpec = runContext.render(this.spec);
+        try {
+            Task task = OBJECT_MAPPER.readValue(taskSpec, Task.class);
+            if (task instanceof TemplatedTask) {
+                throw new IllegalArgumentException("The templated task cannot be of type 'io.kestra.core.tasks.templating.TemplatedTask'");
+            }
+            if (task instanceof RunnableTask<?> runnableTask) {
+                return runnableTask.run(runContext);
+            }
+            throw new IllegalArgumentException("The templated task must be a runnable task");
+        } catch (JsonProcessingException e) {
+            throw new IllegalVariableEvaluationException(e);
+        }
+    }
+}
diff --git a/core/src/main/resources/icons/io.kestra.core.tasks.templating.svg b/core/src/main/resources/icons/io.kestra.core.tasks.templating.svg
new file mode 100644
index 0000000000..42c9dbdd38
--- /dev/null
+++ b/core/src/main/resources/icons/io.kestra.core.tasks.templating.svg
@@ -0,0 +1,2 @@
+<?xml version="1.0" encoding="utf-8"?>
+<svg fill="#000000" width="800px" height="800px" viewBox="0 0 100 100" xmlns="http://www.w3.org/2000/svg"><path d="M69.6,59.3A10.2,10.2,0,1,0,79.8,69.5,10.24,10.24,0,0,0,69.6,59.3Zm6.6,7.4-7.5,7.5a1.45,1.45,0,0,1-2,0L63,70.6a.67.67,0,0,1,0-1l1-1a.67.67,0,0,1,1,0l2.6,2.6,6.5-6.5a.67.67,0,0,1,1,0l1,1A.76.76,0,0,1,76.2,66.7Z"/><path d="M44.5,30H62.3a1.58,1.58,0,0,0,1.6-1.6V25.1A4.91,4.91,0,0,0,59,20.2H47.7a4.91,4.91,0,0,0-4.9,4.9v3.3A1.73,1.73,0,0,0,44.5,30Z"/><path d="M56.1,73.3H31.4a4.89,4.89,0,0,1-4.8-4.9v-34H25a4.89,4.89,0,0,0-4.8,4.9V74.7A4.89,4.89,0,0,0,25,79.6H59.9A12.11,12.11,0,0,1,56.1,73.3Z"/><path d="M35.6,69.5H55.7a15.14,15.14,0,0,1,.7-3.7,13.68,13.68,0,0,1,2.3-4.5H49.3a1.58,1.58,0,0,1-1.6-1.6V58a1.58,1.58,0,0,1,1.6-1.6H65.5c.2,0,.3,0,.4.1a13.61,13.61,0,0,1,3.6-.5A13.89,13.89,0,0,1,76,57.6V29.2a4.91,4.91,0,0,0-4.9-4.9H69.5a.74.74,0,0,0-.8.8v3.3A6.57,6.57,0,0,1,62.2,35H44.5A6.64,6.64,0,0,1,38,28.4V25.1a.74.74,0,0,0-.8-.8H35.6a4.91,4.91,0,0,0-4.9,4.9V64.5A5.06,5.06,0,0,0,35.6,69.5Zm12.1-28a1.58,1.58,0,0,1,1.6-1.6H65.5a1.58,1.58,0,0,1,1.6,1.6v1.6a1.58,1.58,0,0,1-1.6,1.6H49.4a1.58,1.58,0,0,1-1.6-1.6V41.5Zm0,8.2a1.58,1.58,0,0,1,1.6-1.6H65.5a1.58,1.58,0,0,1,1.6,1.6v1.6a1.58,1.58,0,0,1-1.6,1.6H49.4a1.58,1.58,0,0,1-1.6-1.6V49.7Zm-8-8.2a1.58,1.58,0,0,1,1.6-1.6h1.6a1.58,1.58,0,0,1,1.6,1.6v1.6a1.58,1.58,0,0,1-1.6,1.6H41.3a1.58,1.58,0,0,1-1.6-1.6Zm0,8.2a1.58,1.58,0,0,1,1.6-1.6h1.6a1.58,1.58,0,0,1,1.6,1.6v1.6a1.58,1.58,0,0,1-1.6,1.6H41.3a1.58,1.58,0,0,1-1.6-1.6Zm0,8.3a1.58,1.58,0,0,1,1.6-1.6h1.6A1.58,1.58,0,0,1,44.5,58v1.6a1.58,1.58,0,0,1-1.6,1.6H41.3a1.58,1.58,0,0,1-1.6-1.6Z"/></svg>
\ No newline at end of file


Test patch:
diff --git a/core/src/test/java/io/kestra/core/tasks/templating/TemplatedTaskTest.java b/core/src/test/java/io/kestra/core/tasks/templating/TemplatedTaskTest.java
new file mode 100644
index 0000000000..50f6519046
--- /dev/null
+++ b/core/src/test/java/io/kestra/core/tasks/templating/TemplatedTaskTest.java
@@ -0,0 +1,73 @@
+package io.kestra.core.tasks.templating;
+
+import io.kestra.core.models.tasks.Output;
+import io.kestra.core.runners.RunContext;
+import io.kestra.core.runners.RunContextFactory;
+import io.kestra.core.tasks.debugs.Return;
+import io.micronaut.test.extensions.junit5.annotation.MicronautTest;
+import jakarta.inject.Inject;
+import org.junit.jupiter.api.Test;
+
+import java.util.Map;
+
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.instanceOf;
+import static org.hamcrest.Matchers.notNullValue;
+import static org.hamcrest.core.Is.is;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+
+@MicronautTest
+class TemplatedTaskTest {
+
+    @Inject
+    private RunContextFactory runContextFactory;
+
+    @Test
+    void templatedType() throws Exception {
+        RunContext runContext = runContextFactory.of(Map.of("type", "io.kestra.core.tasks.debugs.Return"));
+        TemplatedTask templatedTask = TemplatedTask.builder()
+            .id("template")
+            .type(TemplatedTask.class.getName())
+            .spec("""
+                type: {{ type }}
+                format: It's alive!""")
+            .build();
+
+        Output output = templatedTask.run(runContext);
+
+        assertThat(output, notNullValue());
+        assertThat(output, instanceOf(Return.Output.class));
+        assertThat(((Return.Output)output).getValue(), is("It's alive!"));
+    }
+
+    @Test
+    void templatedFlowable() {
+        RunContext runContext = runContextFactory.of();
+        TemplatedTask templatedTask = TemplatedTask.builder()
+            .id("template")
+            .type(TemplatedTask.class.getName())
+            .spec("""
+                type: io.kestra.core.tasks.flows.Pause
+                delay: PT10S""")
+            .build();
+
+        var exception = assertThrows(IllegalArgumentException.class, () -> templatedTask.run(runContext));
+        assertThat(exception.getMessage(), is("The templated task must be a runnable task"));
+    }
+
+    @Test
+    void templatedTemplated() {
+        RunContext runContext = runContextFactory.of();
+        TemplatedTask templatedTask = TemplatedTask.builder()
+            .id("template")
+            .type(TemplatedTask.class.getName())
+            .spec("""
+                type: io.kestra.core.tasks.templating.TemplatedTask
+                spec: whatever""")
+            .build();
+
+        var exception = assertThrows(IllegalArgumentException.class, () -> templatedTask.run(runContext));
+        assertThat(exception.getMessage(), is("The templated task cannot be of type 'io.kestra.core.tasks.templating.TemplatedTask'"));
+    }
+
+}
\ No newline at end of file