Repository: kestra-io/kestra
Issue: Pebble support for handling YAML data structures (function, filters)
Labels: enhancement
customer-request

### Feature description

With the introduction of [TemplatedTask](https://github.com/kestra-io/kestra/pull/3191) and for typical GitOps engineering tasks, having the possibility to work with YAML inside Kestra tasks would sure come in handy.

Feature proposal, similarily to current behaviour, introduce support for YAML.

### `yaml()` function
Similarily to `json()` [function](https://kestra.io/docs/concepts/expression/function#json) [[code](https://github.com/kestra-io/kestra/blob/develop/core/src/main/java/io/kestra/core/runners/pebble/functions/JsonFunction.java)] support the deserialization / objectification of YAML strings to variables.

### `| yaml` filter
Similarily to `| json` [filter](https://kestra.io/docs/concepts/expression/filter/json#json) [[code](https://github.com/kestra-io/kestra/blob/develop/core/src/main/java/io/kestra/core/runners/pebble/filters/JsonFilter.java)] support the serialization / construction of YAML strings from variables.

### `| yq` filter 
Similarily to `| jq` [filter](https://kestra.io/docs/concepts/expression/filter/json#jq) [[code](https://github.com/kestra-io/kestra/blob/develop/core/src/main/java/io/kestra/core/runners/pebble/filters/JqFilter.java)] enable the use of yq library, which essentially is a mapper for yaml -> json to jq. So there are multiple implementation options.

_Optional workaround would be to parse yaml to an object and then use jq._ Eg. `yaml(inputs.yaml_object) | jq`

### `| indent(n)` string filter
With the use of YAML comes the _issue_ of indentation - especially when constructing YAML from multiple objects separately or having to nest generated YAML within an existing templated structure. Thus I propose a simple filter to apply indentation to strings, that would add `n` number of spaces before each line (except for the first one).

Even better, `indent(n, prefix=" ")`, where `n` is the amount of multiplication of `prefix` that is added before each line in a newline separated string.

NB! Important to note, especially given the examples below in the comment - the first line should *NOT* be indented - this is because with most use-cases we already indent the first line.

Issue comments:
To better illustrate the feature proposal, especially with TemplatedTask and the usage of Kubernetes PodCreate - here is a sample flow, provided runtime configuration and expected task definition that will be run.

### templated-task - flow definition
```
id: templated-task
namespace: kestra.test

labels:
  my-label: "will-be-sent-to-k8s"
  other-label: "value"

inputs:
  # the following represents the object resources with nested structure as supported by Kestra
  # thus they are accessible under the inputs.resources object
  - name: resources.limits.cpu
    type: STRING
    defaults: "1"
    required: true

  - name: resources.limits.memory
    type: STRING
    defaults: "1000Mi"
    required: true

  - name: resources.requests.cpu
    type: STRING
    required: false # note that this is optional and when not filled, it should be skipped in the definition

  - name: resources.requests.memory
    type: STRING
    required: false # note that this is optional and when not filled, it should be skipped in the definition

# other sample using variables, since they allow to explicitly define lists in yaml
variables:
  env: # is presented as a list of objects
    - name: "var_string"
      value: "VALUE"
    - name: "var_int"
      value: 123

tasks:
  - id: dynamic
    type: io.kestra.core.tasks.templating.TemplatedTask
    spec: |
      id: "{{ flow.namespace | slugify }}-{{ flow.id | slugify }}"
      type: io.kestra.plugin.kubernetes.PodCreate
      namespace: kestra
      metadata:
        labels:
          {{ labels | yaml | indent(4) }}
      spec:
        containers:
        - name: debug
          image: alpine:latest
          env:
            {{ variables.env | yaml | indent(6) }}
          command:
            - 'bash'
            - '-c'
            - 'printenv'
        resources:
          {{ inputs.resources | yaml | indent(4) }}
```

### Flow execution context (provided inputs, variables etc.)
A sample run configuration, expressed in YAML with all execution parameters present.
```
flow:
  id: templated-task
  namespace: kestra.test

labels:
  my-label: "will-be-sent-to-k8s"
  other-label: "value"

inputs:
  resources.limits.cpu: "2"
  resources.limits.memory: "1000Mi"
  resources.requests.cpu: "1"
  # resources.requests.memory: # not providing

# other sample using variables, since they allow to explicitly define lists in yaml
variables:
  env: # is presented as a list of objects
    - name: "var_string"
      value: "VALUE"
    - name: "var_int"
      value: 123
```

### Resulting generated task
The following task would be generated and run by TemplatedTask, with the parameters given above.
```
id: "kestra-test-templated-task"
type: io.kestra.plugin.kubernetes.PodCreate
namespace: kestra
metadata:
  labels:
    my-label: "will-be-sent-to-k8s"
    other-label: "value"
spec:
  containers:
  - name: debug
    image: alpine:latest
    env:
      - name: "var_string"
        value: "VALUE"
      - name: "var_int"
        value: 123
    command:
      - 'bash'
      - '-c'
      - 'printenv'
  resources:
    limits:
      cpu: "2"
      memory: "1000Mi"
    requests:
      cpu: "1"
```
@anna-geller , fyi - #3258 can be closed and transferred to milestone `0.16.0` when PR #3283 and PR #3284 are merged to develop.

As for the propsed `| yq` filter, decided not to implement it, since the used Jackson library doesn't incldue YQ and it can easily be mitigated by the following:

```
yaml(yaml_object) | jq(..)
# or
yaml(yaml_object) | jq(..) | yaml # to print out yaml
```

Pull request review comments:

Patch that resolved the issue:
diff --git a/core/src/main/java/io/kestra/core/runners/pebble/Extension.java b/core/src/main/java/io/kestra/core/runners/pebble/Extension.java
index a4602adc91..0782d88059 100644
--- a/core/src/main/java/io/kestra/core/runners/pebble/Extension.java
+++ b/core/src/main/java/io/kestra/core/runners/pebble/Extension.java
@@ -77,6 +77,7 @@ public Map<String, Filter> getFilters() {
         filters.put("flatten",new FlattenFilter());
         filters.put("indent",new IndentFilter());
         filters.put("nindent",new NindentFilter());
+        filters.put("yaml",new YamlFilter());
         return filters;
     }
 
@@ -103,6 +104,7 @@ public Map<String, Function> getFunctions() {
         }
         functions.put("encrypt", new EncryptFunction());
         functions.put("decrypt", new DecryptFunction());
+        functions.put("yaml", new YamlFunction());
 
         return functions;
     }
diff --git a/core/src/main/java/io/kestra/core/runners/pebble/filters/YamlFilter.java b/core/src/main/java/io/kestra/core/runners/pebble/filters/YamlFilter.java
new file mode 100644
index 0000000000..f1f2e945b3
--- /dev/null
+++ b/core/src/main/java/io/kestra/core/runners/pebble/filters/YamlFilter.java
@@ -0,0 +1,55 @@
+package io.kestra.core.runners.pebble.filters;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.SerializationFeature;
+import com.fasterxml.jackson.dataformat.yaml.YAMLFactory;
+import com.fasterxml.jackson.dataformat.yaml.YAMLGenerator;
+import com.fasterxml.jackson.datatype.guava.GuavaModule;
+import com.fasterxml.jackson.datatype.jdk8.Jdk8Module;
+import com.fasterxml.jackson.datatype.jsr310.JavaTimeModule;
+import com.fasterxml.jackson.module.paramnames.ParameterNamesModule;
+import io.pebbletemplates.pebble.error.PebbleException;
+import io.pebbletemplates.pebble.extension.Filter;
+import io.pebbletemplates.pebble.template.EvaluationContext;
+import io.pebbletemplates.pebble.template.PebbleTemplate;
+
+import java.util.List;
+import java.util.Map;
+
+public class YamlFilter implements Filter {
+
+    private static final ObjectMapper MAPPER = new ObjectMapper(
+        new YAMLFactory()
+            .configure(YAMLGenerator.Feature.MINIMIZE_QUOTES, true)
+            .configure(YAMLGenerator.Feature.WRITE_DOC_START_MARKER, false)
+            .configure(YAMLGenerator.Feature.USE_NATIVE_TYPE_ID, false)
+            .configure(YAMLGenerator.Feature.SPLIT_LINES, false)
+            .configure(YAMLGenerator.Feature.INDENT_ARRAYS, true)
+            .configure(YAMLGenerator.Feature.USE_PLATFORM_LINE_BREAKS, true)
+            .configure(YAMLGenerator.Feature.ALWAYS_QUOTE_NUMBERS_AS_STRINGS, false)
+        )
+        .configure(SerializationFeature.WRITE_DATES_AS_TIMESTAMPS, false)
+        .registerModule(new JavaTimeModule())
+        .registerModule(new Jdk8Module())
+        .registerModule(new ParameterNamesModule())
+        .registerModules(new GuavaModule());
+
+    @Override
+    public List<String> getArgumentNames() {
+        return null;
+    }
+
+    @Override
+    public Object apply(Object input, Map<String, Object> args, PebbleTemplate self, EvaluationContext context, int lineNumber) throws PebbleException {
+        if (input == null) {
+            return "null";
+        }
+
+        try {
+            return MAPPER.writeValueAsString(input);
+        } catch (JsonProcessingException e) {
+            throw new PebbleException(e, "Unable to transform to yaml value '" + input +  "' with type '" + input.getClass().getName() + "'", lineNumber, self.getName());
+        }
+    }
+}
diff --git a/core/src/main/java/io/kestra/core/runners/pebble/functions/YamlFunction.java b/core/src/main/java/io/kestra/core/runners/pebble/functions/YamlFunction.java
new file mode 100644
index 0000000000..b5e29d057d
--- /dev/null
+++ b/core/src/main/java/io/kestra/core/runners/pebble/functions/YamlFunction.java
@@ -0,0 +1,49 @@
+package io.kestra.core.runners.pebble.functions;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.JsonMappingException;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.dataformat.yaml.YAMLFactory;
+import com.fasterxml.jackson.dataformat.yaml.snakeyaml.error.YAMLException;
+import io.pebbletemplates.pebble.error.PebbleException;
+import io.pebbletemplates.pebble.extension.Function;
+import io.pebbletemplates.pebble.template.EvaluationContext;
+import io.pebbletemplates.pebble.template.PebbleTemplate;
+
+import java.util.List;
+import java.util.Map;
+
+public class YamlFunction implements Function {
+    final static ObjectMapper MAPPER = new ObjectMapper(
+        new YAMLFactory()
+    ).findAndRegisterModules();
+    private static final TypeReference<Object> TYPE_REFERENCE = new TypeReference<>() {};
+
+    public List<String> getArgumentNames() {
+        return List.of("yaml");
+    }
+
+    @Override
+    public Object execute(Map<String, Object> args, PebbleTemplate self, EvaluationContext context, int lineNumber) {
+        if (!args.containsKey("yaml")) {
+            throw new PebbleException(null, "The 'yaml' function expects an argument 'yaml'.", lineNumber, self.getName());
+        }
+
+        if (!(args.get("yaml") instanceof String)) {
+            throw new PebbleException(null, "The 'yaml' function expects an argument 'yaml' with type string.", lineNumber, self.getName());
+        }
+
+        String yaml = (String) args.get("yaml");
+
+        try {
+            return MAPPER.readValue(yaml, TYPE_REFERENCE);
+        } catch (YAMLException e) {
+            throw new PebbleException(null, "Invalid yaml: " + e.getMessage(), lineNumber, self.getName());
+        } catch (JsonMappingException e) {
+            throw new PebbleException(null, "Invalid yaml: " + e.getMessage(), lineNumber, self.getName());
+        } catch (JsonProcessingException e) {
+            throw new PebbleException(null, "Invalid yaml: " + e.getMessage(), lineNumber, self.getName());
+        }
+    }
+}


Test patch:
diff --git a/core/src/test/java/io/kestra/core/runners/pebble/filters/YamlFilterTest.java b/core/src/test/java/io/kestra/core/runners/pebble/filters/YamlFilterTest.java
new file mode 100644
index 0000000000..bb031ba1dd
--- /dev/null
+++ b/core/src/test/java/io/kestra/core/runners/pebble/filters/YamlFilterTest.java
@@ -0,0 +1,72 @@
+package io.kestra.core.runners.pebble.filters;
+
+import com.google.common.collect.ImmutableMap;
+import io.kestra.core.exceptions.IllegalVariableEvaluationException;
+import io.kestra.core.runners.VariableRenderer;
+import io.micronaut.test.extensions.junit5.annotation.MicronautTest;
+import jakarta.inject.Inject;
+import org.junit.jupiter.api.Test;
+
+import java.time.ZoneId;
+import java.time.ZonedDateTime;
+import java.time.format.DateTimeFormatter;
+import java.util.Arrays;
+import java.util.Map;
+
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.*;
+
+@MicronautTest
+class YamlFilterTest {
+    @Inject
+    VariableRenderer variableRenderer;
+
+    @Test
+    void map() throws IllegalVariableEvaluationException {
+        ZonedDateTime date = ZonedDateTime.parse("2013-09-08T16:19:00+02").withZoneSameLocal(ZoneId.systemDefault());
+
+        ImmutableMap<String, Object> vars = ImmutableMap.of(
+            "vars", ImmutableMap.of("second", Map.of(
+                "string", "string",
+                "int", 1,
+                "float", 1.123F,
+                "list", Arrays.asList(
+                    "string",
+                    1,
+                    1.123F
+                ),
+                "bool", true,
+                "date", date,
+                "map", Map.of(
+                    "string", "string",
+                    "int", 1,
+                    "float", 1.123F
+                )
+            ))
+        );
+
+        String render = variableRenderer.render("{{ vars.second.string | yaml }}", vars);
+        assertThat(render, is("string\n"));
+
+        render = variableRenderer.render("{{ vars.second.int | yaml }}", vars);
+        assertThat(render, is("1\n"));
+
+        render = variableRenderer.render("{{ vars.second.float | yaml }}", vars);
+        assertThat(render, is("1.123\n"));
+
+        render = variableRenderer.render("{{ vars.second.list | yaml }}", vars);
+        assertThat(render, is(" - string\n - 1\n - 1.123\n"));
+
+        render = variableRenderer.render("{{ vars.second.bool | yaml }}", vars);
+        assertThat(render, is("true\n"));
+
+        render = variableRenderer.render("{{ vars.second.date | yaml }}", vars);
+        assertThat(render, is( date.format(DateTimeFormatter.ISO_OFFSET_DATE_TIME) + "\n"));
+
+        render = variableRenderer.render("{{ vars.second.map | yaml }}", vars);
+        assertThat(render, containsString("int: 1\n"));
+        assertThat(render, containsString("int: 1\n"));
+        assertThat(render, containsString("float: 1.123\n"));
+        assertThat(render, containsString("string: string\n"));
+    }
+}
diff --git a/core/src/test/java/io/kestra/core/runners/pebble/functions/YamlFunctionTest.java b/core/src/test/java/io/kestra/core/runners/pebble/functions/YamlFunctionTest.java
new file mode 100644
index 0000000000..e8340f0450
--- /dev/null
+++ b/core/src/test/java/io/kestra/core/runners/pebble/functions/YamlFunctionTest.java
@@ -0,0 +1,84 @@
+package io.kestra.core.runners.pebble.functions;
+
+import com.google.common.collect.ImmutableMap;
+import io.kestra.core.exceptions.IllegalVariableEvaluationException;
+import io.kestra.core.runners.VariableRenderer;
+import io.micronaut.test.extensions.junit5.annotation.MicronautTest;
+import jakarta.inject.Inject;
+import org.junit.jupiter.api.Test;
+
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.is;
+
+@MicronautTest
+class YamlFunctionTest {
+    @Inject
+    VariableRenderer variableRenderer;
+
+    @Test
+    void fromString() throws IllegalVariableEvaluationException {
+
+        ImmutableMap<String, Object> runContext = ImmutableMap.of(
+            "yaml",
+            """
+                # comment
+                string: string
+                int: 123
+                bool: true
+                float: 1.23
+                instant: "1918-02-24T00:00:00Z"
+                date: "1991-08-20"
+                time: "23:59:59"
+                duration: "PT5M6S"
+                'null':
+                object:
+                  key: "value"
+                  child:
+                    key: "value"
+                array:
+                  - string
+                reference: &ref
+                    key: reference
+                default: *ref
+                """
+        );
+
+        String render;
+        render = variableRenderer.render("{{ yaml(yaml).string }}", runContext);
+        assertThat(render, is("string"));
+
+        render = variableRenderer.render("{{ yaml(yaml).int }}", runContext);
+        assertThat(render, is("123"));
+
+        render = variableRenderer.render("{{ yaml(yaml).bool }}", runContext);
+        assertThat(render, is("true"));
+
+        render = variableRenderer.render("{{ yaml(yaml).float }}", runContext);
+        assertThat(render, is("1.23"));
+
+        render = variableRenderer.render("{{ yaml(yaml).instant }}", runContext);
+        assertThat(render, is("1918-02-24T00:00:00Z"));
+
+        render = variableRenderer.render("{{ yaml(yaml).date }}", runContext);
+        assertThat(render, is("1991-08-20"));
+
+        render = variableRenderer.render("{{ yaml(yaml).time }}", runContext);
+        assertThat(render, is("23:59:59"));
+
+        // Kestra internally does not handle null values in objects
+        // render = variableRenderer.render("{{ yaml(yaml).null ?? '' }}", runContext);
+        // assertThat(render, is(""));
+
+        render = variableRenderer.render("{{ yaml(yaml).object.child.key }}", runContext);
+        assertThat(render, is("value"));
+
+        render = variableRenderer.render("{{ yaml(yaml).array[0] }}", runContext);
+        assertThat(render, is("string"));
+
+        // as of 2024-03-15 Jackson YAML does not support anchors
+        // https://github.com/FasterXML/jackson-dataformats-text/issues/98
+        // render = variableRenderer.render("{{ yaml(yaml).default.key }}", runContext);
+        // assertThat(render, is("reference"));
+
+    }
+}