Repository: kestra-io/kestra
Issue: Pebble support for handling YAML data structures (function, filters)
Labels: enhancement
customer-request

### Feature description

With the introduction of [TemplatedTask](https://github.com/kestra-io/kestra/pull/3191) and for typical GitOps engineering tasks, having the possibility to work with YAML inside Kestra tasks would sure come in handy.

Feature proposal, similarily to current behaviour, introduce support for YAML.

### `yaml()` function
Similarily to `json()` [function](https://kestra.io/docs/concepts/expression/function#json) [[code](https://github.com/kestra-io/kestra/blob/develop/core/src/main/java/io/kestra/core/runners/pebble/functions/JsonFunction.java)] support the deserialization / objectification of YAML strings to variables.

### `| yaml` filter
Similarily to `| json` [filter](https://kestra.io/docs/concepts/expression/filter/json#json) [[code](https://github.com/kestra-io/kestra/blob/develop/core/src/main/java/io/kestra/core/runners/pebble/filters/JsonFilter.java)] support the serialization / construction of YAML strings from variables.

### `| yq` filter 
Similarily to `| jq` [filter](https://kestra.io/docs/concepts/expression/filter/json#jq) [[code](https://github.com/kestra-io/kestra/blob/develop/core/src/main/java/io/kestra/core/runners/pebble/filters/JqFilter.java)] enable the use of yq library, which essentially is a mapper for yaml -> json to jq. So there are multiple implementation options.

_Optional workaround would be to parse yaml to an object and then use jq._ Eg. `yaml(inputs.yaml_object) | jq`

### `| indent(n)` string filter
With the use of YAML comes the _issue_ of indentation - especially when constructing YAML from multiple objects separately or having to nest generated YAML within an existing templated structure. Thus I propose a simple filter to apply indentation to strings, that would add `n` number of spaces before each line (except for the first one).

Even better, `indent(n, prefix=" ")`, where `n` is the amount of multiplication of `prefix` that is added before each line in a newline separated string.

NB! Important to note, especially given the examples below in the comment - the first line should *NOT* be indented - this is because with most use-cases we already indent the first line.

Issue comments:
To better illustrate the feature proposal, especially with TemplatedTask and the usage of Kubernetes PodCreate - here is a sample flow, provided runtime configuration and expected task definition that will be run.

### templated-task - flow definition
```
id: templated-task
namespace: kestra.test

labels:
  my-label: "will-be-sent-to-k8s"
  other-label: "value"

inputs:
  # the following represents the object resources with nested structure as supported by Kestra
  # thus they are accessible under the inputs.resources object
  - name: resources.limits.cpu
    type: STRING
    defaults: "1"
    required: true

  - name: resources.limits.memory
    type: STRING
    defaults: "1000Mi"
    required: true

  - name: resources.requests.cpu
    type: STRING
    required: false # note that this is optional and when not filled, it should be skipped in the definition

  - name: resources.requests.memory
    type: STRING
    required: false # note that this is optional and when not filled, it should be skipped in the definition

# other sample using variables, since they allow to explicitly define lists in yaml
variables:
  env: # is presented as a list of objects
    - name: "var_string"
      value: "VALUE"
    - name: "var_int"
      value: 123

tasks:
  - id: dynamic
    type: io.kestra.core.tasks.templating.TemplatedTask
    spec: |
      id: "{{ flow.namespace | slugify }}-{{ flow.id | slugify }}"
      type: io.kestra.plugin.kubernetes.PodCreate
      namespace: kestra
      metadata:
        labels:
          {{ labels | yaml | indent(4) }}
      spec:
        containers:
        - name: debug
          image: alpine:latest
          env:
            {{ variables.env | yaml | indent(6) }}
          command:
            - 'bash'
            - '-c'
            - 'printenv'
        resources:
          {{ inputs.resources | yaml | indent(4) }}
```

### Flow execution context (provided inputs, variables etc.)
A sample run configuration, expressed in YAML with all execution parameters present.
```
flow:
  id: templated-task
  namespace: kestra.test

labels:
  my-label: "will-be-sent-to-k8s"
  other-label: "value"

inputs:
  resources.limits.cpu: "2"
  resources.limits.memory: "1000Mi"
  resources.requests.cpu: "1"
  # resources.requests.memory: # not providing

# other sample using variables, since they allow to explicitly define lists in yaml
variables:
  env: # is presented as a list of objects
    - name: "var_string"
      value: "VALUE"
    - name: "var_int"
      value: 123
```

### Resulting generated task
The following task would be generated and run by TemplatedTask, with the parameters given above.
```
id: "kestra-test-templated-task"
type: io.kestra.plugin.kubernetes.PodCreate
namespace: kestra
metadata:
  labels:
    my-label: "will-be-sent-to-k8s"
    other-label: "value"
spec:
  containers:
  - name: debug
    image: alpine:latest
    env:
      - name: "var_string"
        value: "VALUE"
      - name: "var_int"
        value: 123
    command:
      - 'bash'
      - '-c'
      - 'printenv'
  resources:
    limits:
      cpu: "2"
      memory: "1000Mi"
    requests:
      cpu: "1"
```
@anna-geller , fyi - #3258 can be closed and transferred to milestone `0.16.0` when PR #3283 and PR #3284 are merged to develop.

As for the propsed `| yq` filter, decided not to implement it, since the used Jackson library doesn't incldue YQ and it can easily be mitigated by the following:

```
yaml(yaml_object) | jq(..)
# or
yaml(yaml_object) | jq(..) | yaml # to print out yaml
```

Pull request review comments:

Patch that resolved the issue:
diff --git a/core/src/main/java/io/kestra/core/runners/pebble/AbstractIndent.java b/core/src/main/java/io/kestra/core/runners/pebble/AbstractIndent.java
new file mode 100644
index 0000000000..8c3dd98d48
--- /dev/null
+++ b/core/src/main/java/io/kestra/core/runners/pebble/AbstractIndent.java
@@ -0,0 +1,72 @@
+package io.kestra.core.runners.pebble;
+
+import io.pebbletemplates.pebble.error.PebbleException;
+import io.pebbletemplates.pebble.template.EvaluationContext;
+import io.pebbletemplates.pebble.template.PebbleTemplate;
+
+import java.util.List;
+import java.util.Map;
+
+public abstract class AbstractIndent {
+    public List<String> getArgumentNames() {
+        return List.of("amount", "prefix");
+    }
+
+    protected static String prefix(Map<String, Object> args) {
+        if (args.containsKey("prefix")) {
+            return (String) args.get("prefix");
+        } else {
+            return " ";
+        }
+    }
+
+    protected static String getLineSeparator(String input) {
+        if (input == null)
+            return System.lineSeparator();
+
+        if (input.contains("\r\n"))
+            return "\r\n"; // CRLF
+
+        if (input.contains("\n\r"))
+            return "\n\r"; // LFCR
+
+        if (input.contains("\n"))
+            return "\n"; // LF
+
+        if (input.contains("\r"))
+            return "\r"; // CR
+
+        return System.lineSeparator(); // System default
+    }
+
+    protected Object abstractApply(Object input, Map<String, Object> args, PebbleTemplate self, EvaluationContext context, int lineNumber, String indentType) throws PebbleException {
+        if (input == null) {
+            return null;
+        }
+        if (input.toString().isEmpty()) {
+            return input.toString();
+        }
+
+        if (!args.containsKey("amount")) {
+            throw new PebbleException(null, String.format("The '%s' filter expects an integer as argument 'amount'.", indentType), lineNumber, self.getName());
+        }
+
+        int amount = ((Long) args.get("amount")).intValue();
+        if (!(amount >= 0)) {
+            throw new PebbleException(null, String.format("The '%s' filter expects a positive integer >=0 as argument 'amount'.", indentType), lineNumber, self.getName());
+        }
+
+        String prefix = prefix(args);
+        String newLine = getLineSeparator(input.toString());
+
+        if (indentType.equals("indent"))
+            // indent filter adds N amount of spaces to each line except for the first one (assuming the first line is already indented in place)
+            return input.toString().replace(newLine, newLine + prefix.repeat(amount));
+        else if (indentType.equals("nindent")) {
+            // nindent filter adds a newline to the string and indents each line by defined amount of spaces
+            return (newLine + input).replace(newLine, newLine + prefix.repeat(amount));
+        }
+        throw new PebbleException(null, String.format("Unknow indent type '%s'.", indentType), lineNumber, self.getName());
+
+    }
+}
diff --git a/core/src/main/java/io/kestra/core/runners/pebble/Extension.java b/core/src/main/java/io/kestra/core/runners/pebble/Extension.java
index d9e7faf97d..a4602adc91 100644
--- a/core/src/main/java/io/kestra/core/runners/pebble/Extension.java
+++ b/core/src/main/java/io/kestra/core/runners/pebble/Extension.java
@@ -75,6 +75,8 @@ public Map<String, Filter> getFilters() {
         filters.put("substringAfter", new SubstringAfterFilter());
         filters.put("substringAfterLast", new SubstringAfterLastFilter());
         filters.put("flatten",new FlattenFilter());
+        filters.put("indent",new IndentFilter());
+        filters.put("nindent",new NindentFilter());
         return filters;
     }
 
diff --git a/core/src/main/java/io/kestra/core/runners/pebble/filters/IndentFilter.java b/core/src/main/java/io/kestra/core/runners/pebble/filters/IndentFilter.java
new file mode 100644
index 0000000000..d749724378
--- /dev/null
+++ b/core/src/main/java/io/kestra/core/runners/pebble/filters/IndentFilter.java
@@ -0,0 +1,16 @@
+package io.kestra.core.runners.pebble.filters;
+
+import io.kestra.core.runners.pebble.AbstractIndent;
+import io.pebbletemplates.pebble.error.PebbleException;
+import io.pebbletemplates.pebble.extension.Filter;
+import io.pebbletemplates.pebble.template.EvaluationContext;
+import io.pebbletemplates.pebble.template.PebbleTemplate;
+
+import java.util.Map;
+
+public class IndentFilter extends AbstractIndent implements Filter {
+    @Override
+    public Object apply(Object input, Map<String, Object> args, PebbleTemplate self, EvaluationContext context, int lineNumber) throws PebbleException {
+        return abstractApply(input, args, self, context, lineNumber, "indent");
+    }
+}
diff --git a/core/src/main/java/io/kestra/core/runners/pebble/filters/NindentFilter.java b/core/src/main/java/io/kestra/core/runners/pebble/filters/NindentFilter.java
new file mode 100644
index 0000000000..ae08f2c8d4
--- /dev/null
+++ b/core/src/main/java/io/kestra/core/runners/pebble/filters/NindentFilter.java
@@ -0,0 +1,16 @@
+package io.kestra.core.runners.pebble.filters;
+
+import io.kestra.core.runners.pebble.AbstractIndent;
+import io.pebbletemplates.pebble.error.PebbleException;
+import io.pebbletemplates.pebble.extension.Filter;
+import io.pebbletemplates.pebble.template.EvaluationContext;
+import io.pebbletemplates.pebble.template.PebbleTemplate;
+
+import java.util.Map;
+
+public class NindentFilter extends AbstractIndent implements Filter {
+    @Override
+    public Object apply(Object input, Map<String, Object> args, PebbleTemplate self, EvaluationContext context, int lineNumber) throws PebbleException {
+        return abstractApply(input, args, self, context, lineNumber, "nindent");
+    }
+}


Test patch:
diff --git a/core/src/test/java/io/kestra/core/runners/pebble/filters/IndentFilterTest.java b/core/src/test/java/io/kestra/core/runners/pebble/filters/IndentFilterTest.java
new file mode 100644
index 0000000000..75728baf1c
--- /dev/null
+++ b/core/src/test/java/io/kestra/core/runners/pebble/filters/IndentFilterTest.java
@@ -0,0 +1,79 @@
+package io.kestra.core.runners.pebble.filters;
+
+import io.kestra.core.exceptions.IllegalVariableEvaluationException;
+import io.kestra.core.runners.VariableRenderer;
+import io.micronaut.test.extensions.junit5.annotation.MicronautTest;
+import org.junit.jupiter.api.Test;
+
+import java.util.Map;
+import jakarta.inject.Inject;
+
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.*;
+
+@MicronautTest
+class IndentFilterTest {
+    @Inject
+    VariableRenderer variableRenderer;
+
+    @Test
+    void indentNull() throws IllegalVariableEvaluationException {
+        String render = variableRenderer.render("{{ null | indent(2) }}", Map.of());
+        assertThat(render, emptyOrNullString());
+    }
+
+    @Test
+    void indentEmpty() throws IllegalVariableEvaluationException {
+        String render = variableRenderer.render("{{ '' | indent(2) }}", Map.of());
+        assertThat(render, is(""));
+    }
+
+    @Test
+    void indentEmptyLines() throws IllegalVariableEvaluationException {
+        String render = variableRenderer.render("{{ \"\n\n\" | indent(2) }}", Map.of());
+        assertThat(render, is("\n  \n  "));
+    }
+
+    @Test
+    void indentString() throws IllegalVariableEvaluationException {
+        String render = variableRenderer.render("{{ 'string' | indent(2) }}", Map.of());
+        assertThat(render, is("string"));
+    }
+
+    @Test
+    void indentInteger() throws IllegalVariableEvaluationException {
+        String render = variableRenderer.render("{{ 1 | indent(2) }}", Map.of());
+        assertThat(render, is("1"));
+    }
+
+    @Test
+    void indentStringWithCRLF() throws IllegalVariableEvaluationException {
+        String render = variableRenderer.render("{{ \"first line\r\nsecond line\" | indent(2) }}", Map.of());
+        assertThat(render, is("first line\r\n  second line"));
+    }
+
+    @Test
+    void indentStringWithLF() throws IllegalVariableEvaluationException {
+        String render = variableRenderer.render("{{ \"first line\nsecond line\" | indent(2) }}", Map.of());
+        assertThat(render, is("first line\n  second line"));
+    }
+
+    @Test
+    void indentStringWithCR() throws IllegalVariableEvaluationException {
+        String render = variableRenderer.render("{{ \"first line\rsecond line\" | indent(2) }}", Map.of());
+        assertThat(render, is("first line\r  second line"));
+    }
+
+    @Test
+    void indentStringWithSystemNewLine() throws IllegalVariableEvaluationException {
+        String render = variableRenderer.render("{{ \"first line"+System.lineSeparator()+"second line\" | indent(2) }}", Map.of());
+        assertThat(render, is("first line"+System.lineSeparator()+"  second line"));
+    }
+
+    @Test
+    void indentWithTab() throws IllegalVariableEvaluationException {
+        String render = variableRenderer.render("{{ \"first line\nsecond line\" | indent(2, \"\t\") }}", Map.of());
+        assertThat(render, is("first line\n\t\tsecond line"));
+    }
+
+}
diff --git a/core/src/test/java/io/kestra/core/runners/pebble/filters/NindentFilterTest.java b/core/src/test/java/io/kestra/core/runners/pebble/filters/NindentFilterTest.java
new file mode 100644
index 0000000000..b7ffbe864f
--- /dev/null
+++ b/core/src/test/java/io/kestra/core/runners/pebble/filters/NindentFilterTest.java
@@ -0,0 +1,79 @@
+package io.kestra.core.runners.pebble.filters;
+
+import io.kestra.core.exceptions.IllegalVariableEvaluationException;
+import io.kestra.core.runners.VariableRenderer;
+import io.micronaut.test.extensions.junit5.annotation.MicronautTest;
+import org.junit.jupiter.api.Test;
+
+import java.util.Map;
+import jakarta.inject.Inject;
+
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.*;
+
+@MicronautTest
+class NindentFilterTest {
+    @Inject
+    VariableRenderer variableRenderer;
+
+    @Test
+    void nindentNull() throws IllegalVariableEvaluationException {
+        String render = variableRenderer.render("{{ null | nindent(2) }}", Map.of());
+        assertThat(render, emptyOrNullString());
+    }
+
+    @Test
+    void nindentEmpty() throws IllegalVariableEvaluationException {
+        String render = variableRenderer.render("{{ '' | nindent(2) }}", Map.of());
+        assertThat(render, is(""));
+    }
+
+    @Test
+    void nindentEmptyLines() throws IllegalVariableEvaluationException {
+        String render = variableRenderer.render("{{ \"\n\n\" | nindent(2) }}", Map.of());
+        assertThat(render, is("\n  \n  \n  "));
+    }
+
+    @Test
+    void nindentString() throws IllegalVariableEvaluationException {
+        String render = variableRenderer.render("{{ 'string' | nindent(2) }}", Map.of());
+        assertThat(render, is("\n  string"));
+    }
+
+    @Test
+    void nindentInteger() throws IllegalVariableEvaluationException {
+        String render = variableRenderer.render("{{ 1 | nindent(2) }}", Map.of());
+        assertThat(render, is("\n  1"));
+    }
+
+    @Test
+    void nindentStringWithCRLF() throws IllegalVariableEvaluationException {
+        String render = variableRenderer.render("{{ \"first line\r\nsecond line\" | nindent(2) }}", Map.of());
+        assertThat(render, is("\r\n  first line\r\n  second line"));
+    }
+
+    @Test
+    void nindentStringWithLF() throws IllegalVariableEvaluationException {
+        String render = variableRenderer.render("{{ \"first line\nsecond line\" | nindent(2) }}", Map.of());
+        assertThat(render, is("\n  first line\n  second line"));
+    }
+
+    @Test
+    void nindentStringWithCR() throws IllegalVariableEvaluationException {
+        String render = variableRenderer.render("{{ \"first line\rsecond line\" | nindent(2) }}", Map.of());
+        assertThat(render, is("\r  first line\r  second line"));
+    }
+
+    @Test
+    void nindentStringWithSystemNewLine() throws IllegalVariableEvaluationException {
+        String render = variableRenderer.render("{{ \"first line"+System.lineSeparator()+"second line\" | nindent(2) }}", Map.of());
+        assertThat(render, is(System.lineSeparator()+"  first line"+System.lineSeparator()+"  second line"));
+    }
+
+    @Test
+    void nindentWithTab() throws IllegalVariableEvaluationException {
+        String render = variableRenderer.render("{{ \"first line\nsecond line\" | nindent(2, \"\t\") }}", Map.of());
+        assertThat(render, is("\n\t\tfirst line\n\t\tsecond line"));
+    }
+
+}