Repository: scriban/scriban
Pull request #600: Template code made more customizable
Author: login: julichan
Created: 2025-03-13T21:12:12Z | Merged: 2025-03-13T21:53:48Z

This fixes #599:
- TemplateContext.ResetPreviousNewLine()
- Moved template processing code from IncludeFunction to TemplateContext and made it customizable.

Linked issue #599: Advanced custom function questions
Hello,

I am using Scriban to actually parse database queries and I have a few questions on particular points of scriban code (note that i can make PR myself if you agree with proposed changes):

1. I am implementing alternatives to the include function but it seems a method required to handle new lines properly is not accesible from outside scriban. I have looked around the TemplateContext object but can't find a way to replicate the ResetPreviousNewLine() method. Would you be so kind to make it public or to indicate me an alternative to reproduce the behaviour ?

2. I am caching all parsed templates as Template object, is there a reason it would be bad practice ? If not, it would be great if either of the following changes by order of preference could be made:
 - The ITemplateLoader Load method and LoadAsync method could return an object instead of a string. The value of this is that we could return a precached template or string template and adapt the behaviour in IncludeFunction accordingly. Given the usage, I don't think this would introduce much breaking changes but up to you.
 - The template cache dictionary could be provided as a TemplateContext constructor parameter or we could have a setting to provide existing cache instead of filling elements one by one from another dictionary.

3. Finally, the main question of this post: I would like to build a custom function allowing to input an arbitrary number of blocks as parameters. I'm aware of the wrap function but this only allow for a single block to be passed. I m intending to create my own implementation of necessary interfaces of statements and expression onto the function similarly to ScriptFunction but I'm not certain this is possible nor the right approach. Could you advice on this ?
Exemple of the idea:
```
{% block_function [usual params] %}
  block param 1
{% next %}
  block param 2
{% next %}
  block param 3
{% end %}
```
Thanks for your wise input.
Linked issue comments: user: login: xoofx
created_at: 2025-03-13T13:23:11Z
body: > 1. Would you be so kind to make it public or to indicate me an alternative to reproduce the behaviour ?

You can make a PR, no problem.

> 2\. I am caching all parsed templates as Template object, is there a reason it would be bad practice ? If not, it would be great if either of the following changes by order of preference could be made:
> 
> * The ITemplateLoader Load method and LoadAsync method could return an object instead of a string. The value of this is that we could return a precached template or string template and adapt the behaviour in IncludeFunction accordingly. Given the usage, I don't think this would introduce much breaking changes but up to you.
> * The template cache dictionary could be provided as a TemplateContext constructor parameter or we could have a setting to provide existing cache instead of filling elements one by one from another dictionary.

I would avoid changing `ITemplateLoader` as it would require a breaking change and a bump to a major version.

I would think that moving all this code from IncludeFunction:

https://github.com/scriban/scriban/blob/5184825b47755795aa770ec7622549879010c323/src/Scriban/Functions/IncludeFunction.cs#L71-L103

to something like this would allow to make this part customizable (there are other customization overrides in `TemplateContext`, so that wouldn't be the first)

```c#
partial class TemplateContext 
{
    // ...

    public Template TryGetOrCreateTemplate(string templatePath)
    {
        if (!CachedTemplates.TryGetValue(templatePath, out template))
        {
            CachedTemplates[templatePath] = CreateTemplate(templatePath);
        }
    }

    // can be overriden
    protected virtual Template CreateTemplate(string templatePath)
    {
         // TODO: Add code that was in IncludeFunction
    }
}
```

> Finally, the main question of this post: I would like to build a custom function allowing to input an arbitrary number of blocks as parameters. I'm aware of the wrap function but this only allow for a single block to be passed. I m intending to create my own implementation of necessary interfaces of statements and expression onto the function similarly to ScriptFunction but I'm not certain this is possible nor the right approach. Could you advice on this ?

Blocks (with `{{end}}`) are deep intricated into the parser and generalizing around them might be really difficult.

The "next" block you describe could be not necessarily a `block`, but an intermediate step (could be a function call). Though, It is unclear what do you want to achieve exactly with these "blocks" then. Are they conditionally selected behind? How do you reference them? I'm skeptical that this can be added to Scriban from outside, and so I'm also skeptical about the requirements/feature request (if it is), as it is not a feature that has come up in the past years. The use case (behind it) feels niche/unclear.


author_association: MEMBER
user: login: julichan
created_at: 2025-03-13T17:02:48Z
body: thanks for your quick reply. I ll try to make a PR for point 1 & 2 following your direction. It seems your proposition for point 2 works with me.

As for the next keyword, that was merely a separator between blocks I want to pass to the function so that I can determine what to do with each block. For exemple I wanted a function that handle string.join between blocks but also ensure to ignore empty blocks and eventually allow for a prefix and suffix if at least one block has content.

Since I'm operating with sql queries, it would be useful to make proper statements appear in the query such as 
```
{% include_blocks prefix:"Where " separator:' And ' %}
{% if includeCol1 %}col1 = @fieldName{% end %}
{% next %}
{% if includeCol2 %}col2 = @fieldName2{% end %}
{% end %}
```

I can already do that with a function if I provide template names to load for the fields but that feel overkill to have a field assignment in it's own template. 

Do you have a better way to achieve that?


author_association: CONTRIBUTOR
user: login: julichan
created_at: 2025-03-13T22:06:36Z
body: quick question: on what basis / threshold are you building new version ? thanks
author_association: CONTRIBUTOR
user: login: xoofx
created_at: 2025-03-14T10:18:35Z
body: > quick question: on what basis / threshold are you building new version ? thanks

I'm trying to follow semver 2.0:

- Major: Major/Breaking changes
- Minor: Additional APIs (non breaking)
- Patch: Simple bug fixes

Ideally, Scriban should use [PublicApiAnalyzers](https://github.com/dotnet/roslyn-analyzers/blob/main/src/PublicApiAnalyzers/PublicApiAnalyzers.Help.md) but with the particular setup of the project with nullable that is half implemented, it feels difficult to make it working with it.
author_association: MEMBER
user: login: julichan
created_at: 2025-04-01T05:55:32Z
body: sorry, my question wasn't clear. What I wanted to know is how frequently you make a release. But it may not be a frequency but based on a quantity of changes or something. My point was to know when to expect the update to come as a nuget. Thanks
author_association: CONTRIBUTOR
user: login: xoofx
created_at: 2025-04-01T05:59:51Z
body: > sorry, my question wasn't clear. What I wanted to know is how frequently you make a release. But it may not be a frequency but based on a quantity of changes or something. My point was to know when to expect the update to come as a nuget. Thanks

Oh, sometimes when someone asks 😅 

I was waiting for a few tweaks landing in the past days. But I can release a new version now I think.
author_association: MEMBER
user: login: julichan
created_at: 2025-04-01T06:06:47Z
body: Please thanks. I did not want to pressure you. I might bring more improvements later if I feel my "feature" is actually useful.
author_association: CONTRIBUTOR
user: login: julichan
created_at: 2025-04-01T06:09:13Z
body: Also one more question. I intent to bring out my framework as open-source on github when it's mature enough. Think it s possible we reference each other on Readme later? 
author_association: CONTRIBUTOR
user: login: xoofx
created_at: 2025-04-01T06:20:04Z
body: > Also one more question. I intent to bring out my framework as open-source on github when it's mature enough. Think it s possible we reference each other on Readme later?

Sure, there is a section in the readme for such links: https://github.com/scriban/scriban?tab=readme-ov-file#related-projects
author_association: MEMBER
user: login: xoofx
created_at: 2025-04-01T06:21:27Z
body: > Please thanks. I did not want to pressure you. I might bring more improvements later if I feel my "feature" is actually useful.

`6.1.0` just got released
author_association: MEMBER

Diff:
diff --git a/src/Scriban/Functions/IncludeFunction.cs b/src/Scriban/Functions/IncludeFunction.cs
index b5c3dc3e..7e3ea2f0 100644
--- a/src/Scriban/Functions/IncludeFunction.cs
+++ b/src/Scriban/Functions/IncludeFunction.cs
@@ -68,39 +68,7 @@ public object Invoke(TemplateContext context, ScriptNode callerContext, ScriptAr
                 throw new ScriptRuntimeException(callerContext.Span, $"Include template path is null for `{templateName}");
             }
 
-            Template template;
-
-            if (!context.CachedTemplates.TryGetValue(templatePath, out template))
-            {
-
-                string templateText;
-                try
-                {
-                    templateText = templateLoader.Load(context, callerContext.Span, templatePath);
-                }
-                catch (Exception ex) when (!(ex is ScriptRuntimeException))
-                {
-                    throw new ScriptRuntimeException(callerContext.Span, $"Unexpected exception while loading the include `{templateName}` from path `{templatePath}`", ex);
-                }
-
-                if (templateText == null)
-                {
-                    throw new ScriptRuntimeException(callerContext.Span, $"The result of including `{templateName}->{templatePath}` cannot be null");
-                }
-
-                // Clone parser options
-                var parserOptions = context.TemplateLoaderParserOptions;
-                var lexerOptions = context.TemplateLoaderLexerOptions;
-                template = Template.Parse(templateText, templatePath, parserOptions, lexerOptions);
-
-                // If the template has any errors, throw an exception
-                if (template.HasErrors)
-                {
-                    throw new ScriptParserRuntimeException(callerContext.Span, $"Error while parsing template `{templateName}` from `{templatePath}`", template.Messages);
-                }
-
-                context.CachedTemplates.Add(templatePath, template);
-            }
+            Template template = context.GetOrCreateTemplate(templatePath, callerContext);
 
             // Make sure that we cannot recursively include a template
             object result = null;
diff --git a/src/Scriban/TemplateContext.cs b/src/Scriban/TemplateContext.cs
index 6d606556..8e19ca1d 100644
--- a/src/Scriban/TemplateContext.cs
+++ b/src/Scriban/TemplateContext.cs
@@ -1203,10 +1203,50 @@ protected virtual IListAccessor GetListAccessorImpl(object target, Type type)
             return null;
         }
 
-        internal void ResetPreviousNewLine()
+        public void ResetPreviousNewLine()
         {
             _previousTextWasNewLine = false;
         }
+
+        public Template GetOrCreateTemplate(string templatePath, ScriptNode callerContext)
+        {
+            if (!CachedTemplates.TryGetValue(templatePath, out var template))
+            {
+                template = CreateTemplate(templatePath, callerContext);
+                CachedTemplates[templatePath] = template;
+            }
+            return template;
+        }
+
+        protected virtual Template CreateTemplate(string templatePath, ScriptNode callerContext)
+        {
+            string templateText;
+            try
+            {
+                templateText = TemplateLoader.Load(this, callerContext.Span, templatePath);
+            }
+            catch (Exception ex) when (!(ex is ScriptRuntimeException))
+            {
+                throw new ScriptRuntimeException(callerContext.Span, $"Unexpected exception while creating template from path `{templatePath}`", ex);
+            }
+
+            if (templateText == null)
+            {
+                throw new ScriptRuntimeException(callerContext.Span, $"The result of including `{templatePath}` cannot be null");
+            }
+
+            var template = Template.Parse(templateText, templatePath, TemplateLoaderParserOptions, TemplateLoaderLexerOptions);
+
+            // If the template has any errors, throw an exception
+            if (template.HasErrors)
+            {
+                throw new ScriptParserRuntimeException(callerContext.Span, $"Error while parsing template `{templatePath}`", template.Messages);
+            }
+
+            CachedTemplates.Add(templatePath, template);
+
+            return template;
+        }
     }
 
     /// <summary>


Reviews:
user: login: julichan
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-13T21:16:41Z
user: login: julichan
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-13T21:19:00Z
user: login: xoofx
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-13T21:19:59Z
user: login: julichan
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-13T21:21:52Z
user: login: julichan
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-13T21:23:22Z
user: login: xoofx
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-13T21:24:51Z
user: login: julichan
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-13T21:29:49Z
user: login: julichan
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-13T21:37:57Z
user: login: xoofx
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-13T21:40:43Z
user: login: julichan
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-13T21:43:05Z
user: login: julichan
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-13T21:45:39Z

Review comments:
diff_hunk: @@ -68,39 +68,7 @@ public object Invoke(TemplateContext context, ScriptNode callerContext, ScriptAr
                 throw new ScriptRuntimeException(callerContext.Span, $"Include template path is null for `{templateName}");
             }
 
-            Template template;
-
-            if (!context.CachedTemplates.TryGetValue(templatePath, out template))
-            {
-
-                string templateText;
-                try
-                {
-                    templateText = templateLoader.Load(context, callerContext.Span, templatePath);
-                }
-                catch (Exception ex) when (!(ex is ScriptRuntimeException))
-                {
-                    throw new ScriptRuntimeException(callerContext.Span, $"Unexpected exception while loading the include `{templateName}` from path `{templatePath}`", ex);
-                }
-
-                if (templateText == null)
-                {
-                    throw new ScriptRuntimeException(callerContext.Span, $"The result of including `{templateName}->{templatePath}` cannot be null");
-                }
-
-                // Clone parser options
-                var parserOptions = context.TemplateLoaderParserOptions;
-                var lexerOptions = context.TemplateLoaderLexerOptions;
-                template = Template.Parse(templateText, templatePath, parserOptions, lexerOptions);
-
-                // If the template has any errors, throw an exception
-                if (template.HasErrors)
-                {
-                    throw new ScriptParserRuntimeException(callerContext.Span, $"Error while parsing template `{templateName}` from `{templatePath}`", template.Messages);
-                }
-
-                context.CachedTemplates.Add(templatePath, template);
-            }
+            Template template = context.GetOrProcessTemplate(templatePath, callerContext);
 
             // Make sure that we cannot recursively include a template
             object result = null;
path: src/Scriban/Functions/IncludeFunction.cs
user: login: julichan
body: This is not this PR subject but I could move this code into a TemplateContext's method as well to make this part callable from other objects. I'd reuse that part of code in several alternative include function if it was available but up to you.
```
            object result = null;
            context.EnterRecursive(callerContext);
            var previousIndent = context.CurrentIndent;
            context.CurrentIndent = null;
            context.PushOutput();
            var previousArguments = context.GetValue(ScriptVariable.Arguments);
            try
            {
                context.SetValue(ScriptVariable.Arguments, arguments, true, true);
                if (previousIndent != null)
                {
                    // We reset before and after the fact that we have a new line
                    context.ResetPreviousNewLine();
                }
                result = template.Render(context);
                if (previousIndent != null)
                {
                    context.ResetPreviousNewLine();
                }
            }
            finally
            {
                context.PopOutput();
                context.CurrentIndent = previousIndent;
                context.ExitRecursive(callerContext);
                // Remove the arguments
                context.DeleteValue(ScriptVariable.Arguments);
                if (previousArguments != null)
                {
                    // Restore them if necessary
                    context.SetValue(ScriptVariable.Arguments, previousArguments, true);
                }
            }
```
created_at: 2025-03-13T21:16:41Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1203,10 +1203,50 @@ protected virtual IListAccessor GetListAccessorImpl(object target, Type type)
             return null;
         }
 
-        internal void ResetPreviousNewLine()
+        public void ResetPreviousNewLine()
         {
             _previousTextWasNewLine = false;
         }
+
+        public Template GetOrProcessTemplate(string templatePath, ScriptNode callerContext)
+        {
+            if (!CachedTemplates.TryGetValue(templatePath, out var template))
+            {
+                template = ProcessTemplate(templatePath, callerContext);
+                CachedTemplates[templatePath] = template;
+            }
+            return template;
+        }
+
+        protected virtual Template ProcessTemplate(string templatePath, ScriptNode callerContext)
path: src/Scriban/TemplateContext.cs
user: login: julichan
body: I chose the word Process rather than Create because Create did not feel like the right action since it's loading a template from a source and parsing a string into a Template objects. Overrides may not even go that way. I could not find something less general than Process but this feels fitting to me. What do you think ?
created_at: 2025-03-13T21:19:00Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1203,10 +1203,50 @@ protected virtual IListAccessor GetListAccessorImpl(object target, Type type)
             return null;
         }
 
-        internal void ResetPreviousNewLine()
+        public void ResetPreviousNewLine()
         {
             _previousTextWasNewLine = false;
         }
+
+        public Template GetOrProcessTemplate(string templatePath, ScriptNode callerContext)
path: src/Scriban/TemplateContext.cs
user: login: xoofx
body: I think I prefer the name `GetOrCreateTemplate`. `Process` might imply that it is running it or something more than just creating a Template object.
created_at: 2025-03-13T21:19:59Z
author_association: MEMBER
diff_hunk: @@ -1203,10 +1203,50 @@ protected virtual IListAccessor GetListAccessorImpl(object target, Type type)
             return null;
         }
 
-        internal void ResetPreviousNewLine()
+        public void ResetPreviousNewLine()
         {
             _previousTextWasNewLine = false;
         }
+
+        public Template GetOrProcessTemplate(string templatePath, ScriptNode callerContext)
path: src/Scriban/TemplateContext.cs
user: login: julichan
body: I made the exact opposite comment with an explanation of my choice on another line. haha.
created_at: 2025-03-13T21:21:52Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1203,10 +1203,50 @@ protected virtual IListAccessor GetListAccessorImpl(object target, Type type)
             return null;
         }
 
-        internal void ResetPreviousNewLine()
+        public void ResetPreviousNewLine()
         {
             _previousTextWasNewLine = false;
         }
+
+        public Template GetOrProcessTemplate(string templatePath, ScriptNode callerContext)
path: src/Scriban/TemplateContext.cs
user: login: julichan
body: To process something refers to a series of actions or steps taken to achieve a specific outcome. This can apply to various contexts, such as converting raw materials into a finished product, handling data, or executing a business operation. In a business context, a process definition is a detailed description of a process, including its start transition and state definitions, expressed through finite guarded recursive specifications.
created_at: 2025-03-13T21:23:22Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1203,10 +1203,50 @@ protected virtual IListAccessor GetListAccessorImpl(object target, Type type)
             return null;
         }
 
-        internal void ResetPreviousNewLine()
+        public void ResetPreviousNewLine()
         {
             _previousTextWasNewLine = false;
         }
+
+        public Template GetOrProcessTemplate(string templatePath, ScriptNode callerContext)
+        {
+            if (!CachedTemplates.TryGetValue(templatePath, out var template))
+            {
+                template = ProcessTemplate(templatePath, callerContext);
+                CachedTemplates[templatePath] = template;
+            }
+            return template;
+        }
+
+        protected virtual Template ProcessTemplate(string templatePath, ScriptNode callerContext)
path: src/Scriban/TemplateContext.cs
user: login: xoofx
body: Ahah, I said the opposite below. 😅 

I think that Create is easier to reason about. Process almost implies that it is going to render it. This method returns a Template, so it fits well with the generic term "Create".
created_at: 2025-03-13T21:24:51Z
author_association: MEMBER
diff_hunk: @@ -1203,10 +1203,50 @@ protected virtual IListAccessor GetListAccessorImpl(object target, Type type)
             return null;
         }
 
-        internal void ResetPreviousNewLine()
+        public void ResetPreviousNewLine()
         {
             _previousTextWasNewLine = false;
         }
+
+        public Template GetOrProcessTemplate(string templatePath, ScriptNode callerContext)
+        {
+            if (!CachedTemplates.TryGetValue(templatePath, out var template))
+            {
+                template = ProcessTemplate(templatePath, callerContext);
+                CachedTemplates[templatePath] = template;
+            }
+            return template;
+        }
+
+        protected virtual Template ProcessTemplate(string templatePath, ScriptNode callerContext)
path: src/Scriban/TemplateContext.cs
user: login: julichan
body: May I argue on this, I posted the definition of to process on your post but if you still disagree, I won't oppose to make the change. 🤣 
created_at: 2025-03-13T21:29:49Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1203,10 +1203,50 @@ protected virtual IListAccessor GetListAccessorImpl(object target, Type type)
             return null;
         }
 
-        internal void ResetPreviousNewLine()
+        public void ResetPreviousNewLine()
         {
             _previousTextWasNewLine = false;
         }
+
+        public Template GetOrProcessTemplate(string templatePath, ScriptNode callerContext)
+        {
+            if (!CachedTemplates.TryGetValue(templatePath, out var template))
+            {
+                template = ProcessTemplate(templatePath, callerContext);
+                CachedTemplates[templatePath] = template;
+            }
+            return template;
+        }
+
+        protected virtual Template ProcessTemplate(string templatePath, ScriptNode callerContext)
path: src/Scriban/TemplateContext.cs
user: login: julichan
body: Sorry, I'm being picky about the choice of word of a function I provide. I can propose the action "Prepare" instead as well if Process really displease you. 😆 
created_at: 2025-03-13T21:37:57Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1203,10 +1203,50 @@ protected virtual IListAccessor GetListAccessorImpl(object target, Type type)
             return null;
         }
 
-        internal void ResetPreviousNewLine()
+        public void ResetPreviousNewLine()
         {
             _previousTextWasNewLine = false;
         }
+
+        public Template GetOrProcessTemplate(string templatePath, ScriptNode callerContext)
path: src/Scriban/TemplateContext.cs
user: login: xoofx
body: Yeah, naming is hard and while it's debatable, it can be like colors 😁. This function just creates a template. It does not process it. The default implementation parses a text representation to return a Template instance, another implementation could just return an instance from a pool of templates. In the end it creates an instance to put in the cache, and this is usually what people think about in the .NET ecosystem (e.g. [CacheExtensions.GetOrCreate](https://learn.microsoft.com/en-us/dotnet/api/microsoft.extensions.caching.memory.cacheextensions.getorcreate))
created_at: 2025-03-13T21:40:43Z
author_association: MEMBER
diff_hunk: @@ -1203,10 +1203,50 @@ protected virtual IListAccessor GetListAccessorImpl(object target, Type type)
             return null;
         }
 
-        internal void ResetPreviousNewLine()
+        public void ResetPreviousNewLine()
         {
             _previousTextWasNewLine = false;
         }
+
+        public Template GetOrProcessTemplate(string templatePath, ScriptNode callerContext)
path: src/Scriban/TemplateContext.cs
user: login: julichan
body: ok you made your point. I'll make an update instead of stalling on this 😄 
created_at: 2025-03-13T21:43:05Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1203,10 +1203,50 @@ protected virtual IListAccessor GetListAccessorImpl(object target, Type type)
             return null;
         }
 
-        internal void ResetPreviousNewLine()
+        public void ResetPreviousNewLine()
         {
             _previousTextWasNewLine = false;
         }
+
+        public Template GetOrProcessTemplate(string templatePath, ScriptNode callerContext)
path: src/Scriban/TemplateContext.cs
user: login: julichan
body: done
created_at: 2025-03-13T21:45:39Z
author_association: CONTRIBUTOR

PR comments:
user: login: xoofx
created_at: 2025-03-13T21:53:57Z
body: Thanks!
author_association: MEMBER
user: login: julichan
created_at: 2025-03-13T21:54:21Z
body: Hell, no! Thanks to you @xoofx !!!!
author_association: CONTRIBUTOR