Repository: netbox-community/netbox
Pull request: Fixes #17490: Config Template unable to dynamically include templates
### Fixes: #17490

Config templates are unable to dynamically include child templates.  This is a proposed fix as suggested by @ed-ud in #18066 which was closed as a duplicate. 

Problem statement:
Config Template unable to dynamically include templates
### Deployment Type

Self-hosted

### NetBox Version

v4.0.9

### Python Version

3.10

### Steps to Reproduce

1. Create the following template in a data store, as well as child templates in the blocks directory.  In this example a template named interface.j2 is located in the blocks/TEST/ directory.  
2. Assign the template to a device
3. Attempt to render the template

Example of template.  This has been tested manually using Jinja2. The configuration blocks referenced are from a plugin.  

config system interface
{% for config_block in device.site.configblock_set.all() %}
{% include "blocks/" + custom_block.name + "/interface.j2" %}
{% endfor %}
end

### Expected Behavior

Expected behaviour is that the template renders and imports the relevant includes. 

### Observed Behavior

Traceback (most recent call last):
  File "/opt/netbox/netbox/utilities/jinja2.py", line 26, in get_source
    template_source = self._template_cache[template]
KeyError: 'blocks/TEST/interface.j2'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/netbox/netbox/dcim/views.py", line 2060, in get_extra_context
    rendered_config = config_template.render(context=context_data)
  File "/opt/netbox/netbox/extras/models/configs.py", line 282, in render
    output = template.render(**_context)
  File "/usr/local/lib/python3.10/site-packages/jinja2/environment.py", line 1304, in render
    self.environment.handle_exception()
  File "/usr/local/lib/python3.10/site-packages/jinja2/environment.py", line 939, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "block-config.j2", line 49, in top-level template code
  File "/opt/netbox/netbox/utilities/jinja2.py", line 28, in get_source
    raise TemplateNotFound(template)
jinja2.exceptions.TemplateNotFound: blocks/TEST/interface.j2

Hints from issue discussion:
Tested this and confirmed this does indeed not import the template.

I honestly have no idea what could be going wrong here as most of the jinja rendering happens within jinja itself.  This may be an upstream bug but I am not sure.
Thanks for checking.  It's very odd alright.  I have unittests on the Gitlab repo which hosts the config template, which renders the templates using some test data.  It works perfectly on there.  

I had a look at the Netbox code and suspect it might could have something to do with the jinja2 sandbox being used but I'm really not sure.  I'll have to do some more testing to emulate it as closely as possible.

Commit message: Fixes #17490: Config Template unable to dynamically include templates

Patch to review:
diff --git a/netbox/utilities/jinja2.py b/netbox/utilities/jinja2.py
index cefb97831d9..8286ae17130 100644
--- a/netbox/utilities/jinja2.py
+++ b/netbox/utilities/jinja2.py
@@ -29,10 +29,16 @@ def get_source(self, environment, template):
 
         # Find and pre-fetch referenced templates
         if referenced_templates := find_referenced_templates(environment.parse(template_source)):
-            self.cache_templates({
-                df.path: df.data_as_string for df in
-                DataFile.objects.filter(source=self.data_source, path__in=referenced_templates)
-            })
+            if None in referenced_templates:
+                self.cache_templates({
+                    df.path: df.data_as_string for df in
+                    DataFile.objects.filter(source=self.data_source)
+                })
+            else:
+                self.cache_templates({
+                    df.path: df.data_as_string for df in
+                    DataFile.objects.filter(source=self.data_source, path__in=referenced_templates)
+                })
 
         return template_source, template, lambda: True
 


Code review comments:
diff_hunk: @@ -29,10 +29,16 @@ def get_source(self, environment, template):
 
         # Find and pre-fetch referenced templates
         if referenced_templates := find_referenced_templates(environment.parse(template_source)):
path: netbox/utilities/jinja2.py
text: @user1:
There's a very odd little bug hidden here when not using dynamic resolution for related templates. We need to cast the generator returned by `find_referenced_templates()` to an iterable to avoid exhausting it on the check for None. (Otherwise it ends up empty when passed as the value for `path__in`.)
```suggestion
        if referenced_templates := tuple(find_referenced_templates(environment.parse(template_source))):
```

@user2:
Nice catch!

diff_hunk: @@ -29,10 +29,16 @@ def get_source(self, environment, template):
 
         # Find and pre-fetch referenced templates
         if referenced_templates := find_referenced_templates(environment.parse(template_source)):
-            self.cache_templates({
-                df.path: df.data_as_string for df in
-                DataFile.objects.filter(source=self.data_source, path__in=referenced_templates)
-            })
+            if None in referenced_templates:
+                self.cache_templates({
+                    df.path: df.data_as_string for df in
+                    DataFile.objects.filter(source=self.data_source)
+                })
+            else:
+                self.cache_templates({
+                    df.path: df.data_as_string for df in
+                    DataFile.objects.filter(source=self.data_source, path__in=referenced_templates)
+                })
path: netbox/utilities/jinja2.py
text: @user1:
We can conditionally apply the `path__in` filter to avoid duplicating code. We should also explain the significance of None in the paths list as its meaning is not obvious IMO.
```suggestion
            related_files = DataFile.objects.filter(source=self.data_source)
            # None indicates the use of dynamic resolution. If dependent files are statically
            # defined, we can filter by path for optimization.
            if None not in referenced_templates:
                related_files = related_files.filter(path__in=referenced_templates)
            self.cache_templates({
                df.path: df.data_as_string for df in related_files
            })
```

@user2:
I wonder if this code-dedup trick has the side-effect of fetching/caching ALL the files for a data source even when dynamic includes are NOT in use?  I had originally written it to work with the original optimization, and then resort to loading ALL the files only when necessary.  It may not matter... you know the code better than I do... :-)

@user1:
Django querysets are lazy; they won't actually trigger a database query until evaluation has been forced (when the dict is generated).