Package: j2gpp
J2GPP - Jinja2-based General Purpose Preprocessorj2gppis a command-line tool for rendering templates using the Jinja2 syntax. It's intended purpose is to serve as a preprocessor for any programming or markup language with a unified syntax and flow across languages.J2GPP - Jinja2-based General Purpose PreprocessorInstallationBasic usageCommand line argumentsSpecify output directorySpecifying output fileInclude search directoryPassing global variables in command lineLoading global variables from filesLoading global variables from environmentLoading custom Jinja2 filtersLoading custom Jinja2 testsProcessing variables before renderingOption flagsContext variablesBuilt-in filtersProcess directoriesSupported formats for variablesCommand line defineYAMLJSONXMLTOMLINI/CFGENVCSV/TSVScripting in J2GPP templatesConditional and do extensionConditional and filtersThrow errors and warnings from templateWriting filesWrite exampleAppend examplePath argumentWriting to both filesSkipping the parent fileInstallationWithPython >= 3.7, Install from Pypi :pipinstallj2gppBasic usagej2gpprequires at least one source be provided. The source paths can be files or directories, relative or absolute, and can use UNIX-style patterns such as wildcards. Template file names must end with the.j2extension which will be stripped at render.For more information about the Jinja2 syntax, see the documentation atjinja.palletsprojects.com.For instance, suppose we have a templatized source filefoo.c.j2:#include<stdio.h>{%setmessage="Hello, world!"%}intmain(){printf("{{message}}");return0;}We can render the template usingj2gpp:j2gpp./foo.c.j2The output is written tofoo.cnext to the source template :#include<stdio.h>intmain(){printf("Hello, world!");return0;}The following arguments can be added to the command for additional features. The details of each command is explained in the sections below.ArgumentDescription-O/--outdirOutput directory for all rendered templates-o/--outputOutput file for single template-I/--incdirInclude search directory for include and import statements-D/--defineInline global variables for all templates-V/--varfileGlobal variables files for all templates--envvarLoads environment variables as global variables--filtersLoad extra Jinja2 filters from a Python file--testsLoad extra Jinja2 tests from a Python file--file-vars-adapterLoad a Python function to process variables after loading from a file--global-vars-adapterLoad a Python function to process all variables before rendering--overwrite-outdirOverwrite output directory--warn-overwriteWarn when overwriting files--no-overwritePrevent overwriting files--no-strict-undefinedDisable error with undefined variable in template--no-check-identifierDisable warning when attributes are not valid identifiers--fix-identifiersReplace invalid characters from identifiers with underscore--csv-delimiterCSV delimiter (default: ',')--csv-escapecharCSV escape character (default: None)--csv-dontstripDisable stripping whitespace of CSV values--render-non-templateProcess also source files that are not recognized as templates--copy-non-templateCopy source files that are not templates to output directory--force-globGlob UNIX-like patterns in path even when quoted--perfMeasure the execution time for performance testing--versionPrint J2GPP version and quits--licensePrint J2GPP license and quitsCommand line argumentsSpecify output directoryBy default the rendered files are saved next to the source templates. You can provide an output directory with the-O/--outdirargument. The output directory path can be relative or absolute. If the directory doesn't exist, it will be created.For instance the following command will write the rendered file to./bar/foo.c.j2gpp./foo.c.j2--outdir./bar/Specifying output fileBy default the rendered files are saved next to the source templates. If a single source template is provided, you can specify the output file directory and name with the-o/--outputargument. The output file path can be relative or absolute. If the directory doesn't exist, it will be created.For instance the following command will write the rendered file to./bar.c.j2gpp./foo.c.j2--output./bar.cInclude search directoryTheincludeandimportJinja2 statements require specifying the directory in which the renderer will search. That is provided using the-I/--incidrargument.For instance, with the following command, the files in the directory./includes/will be available toincludeandimportstatements when rendering the templatefoo.c.j2.j2gpp./foo.c.j2--incdir./includes/Passing global variables in command lineYou can pass global variables to all templates rendered using the-D/--defineargument with a list of variables in the formatname=value. Values are parsed to cast to the correct Python type as explainedlater. Dictionary attributes to any depth can be assigned using dots "." to separate the keys. Global variables defined in the command line overwrite the global variables set by loading files.For instance, with the following command, the variablebarwill have the value42when rendering the templatefoo.c.j2.j2gpp./foo.c.j2--definebar=42Loading global variables from filesYou can load global variables from files using the-V/--varfileargument with a list of files. The file paths can be relative or absolute, and can use UNIX-style patterns such as wildcards. Variables file types supported right now are YAML, JSON, XML, TOML, INI/CFG, ENV, CSV and TSV. Global variables loaded from files are overwritten by variables defined in the command line.For instance, with the following command, the variablebarwill have the value42when rendering the templatefoo.c.j2.j2gpp./foo.c.j2--varfile./qux.ymlWith the variables filequx.yml:bar: 42Loading global variables from environmentYou can import the environment variables of the shell as global variables using the--envvarargument. The name of the variables will be that of the environment variable and the value will be cast automatically to the proper Python/Jinja2 type.For instance, with the following command, the variableBARwill have the value42when rendering the templatefoo.c.j2.exportBAR=42j2gpp./foo.c.j2--envvarIf a string is provided after the--envvarargument, the environment variables will be stored in an object of the name provided instead of at the root.For instance, with the following command, the variableENV.BARwill have the value42when rendering the templatefoo.c.j2.exportBAR=42j2gpp./foo.c.j2--envvarENVLoading custom Jinja2 filtersYou can import custom Jinja2 filters by providing Python files with the--filtersargument. All functions defined in the python files will be available as Jinja2 filters in the templates.For instance, with the following command and python file, the filterright_ajustwill be available when rendering the templatefoo.c.j2.j2gpp./foo.c.j2--filters./bar.py# bar.pydefright_ajust(s,length=0):returns.rjust(length)Loading custom Jinja2 testsYou can import custom Jinja2 tests by providing Python files with the--testsargument. All functions defined in the python files will be available as Jinja2 tests in the templates.For instance, with the following command and python file, the testprimewill be available when rendering the templatefoo.c.j2.j2gpp./foo.c.j2--tests./bar.py# bar.pyimportmathdefprime(x):ifx<=1:returnFalseforiinrange(2,int(math.sqrt(x))+1):if(x%i)==0:returnFalsereturnTrueProcessing variables before renderingYou can perform transformations on the dictionary storing the variables before rendering templates by providing a Python function with the argument--global-vars-adapter. It takes two arguments, the first is the path to the Python script, and the second is the name of the function to use. The function must take in a single argument, the variables dictionary, and modify it as a reference (not return the modified dictionary). You can also provide an adapter function to be run on the variables loaded from each file before they are written to the global variables object by providing a function with the argument--file-vars-adapter.For instance, with the following command and python file, the variable dictionary loaded from the filequx.ymlwill be processed by the functionshout_values()before rendering the templatefoo.c.j2.j2gpp./foo.c.j2--varfile./qux.yml--file-vars-adapter./bar.pyshout_values# bar.pydefshout_values(var_dict):forkey,valinvar_dict.items():ifisinstance(val,str):var_dict[key]=val.upper()Option flagsSome arguments are flags to enable or disable special features. This is more advanced but can be useful in niche situations.--overwrite-outdircleans the output directory before rendering the templates and copying any other files.--warn-overwriteenables warnings triggered whenever a file is overwritten.--no-overwriteprevents any file from being overwritten, and triggers a warning when that happens.--no-strict-undefineddisables errors triggered whenever a template variable is used but not defined.--no-check-identifierdisables the ckecking that the variables names and attributes are valid Python identifiers. Root variables with a name not passing this check will not be accessible in Jinja2 templates.--fix-identifiersfixes variables and attributes names that are not valid Python identifiers by replacing incorrect characters by underscores, and if the first character is a number, an underscore is added before.--csv-delimiterfollowed by a string will change the delimiter used to parse CSV variables files. The default is ",".--csv-escapecharfollowed by a character sets the escape character used to parse CSV variables files. There is no escape character by default.--csv-dontstripdisables the stripping of whitespace from CSV keys and values.--render-non-templateforces every source file found to be rendered, even if they are not recognized as a template (by ending with a template extension). The resulting file will be saved in the location following the rules of regular templates, but instead of removing the template extension, they will have a suffix added before the file extensions. By default, this suffix is_j2gpp, but this can be replaced by whatever is specified after the flag argument.--copy-non-templateenables the copying of the source files that are not recognized as templates or the files in the source directories to the output directory when one is provided with the--outdirargument.--force-globenables globbing UNIX-like patterns in the source files paths even if they are surrounded by quotes. This is disabled by default to allow processing files with*and[...]in their path. Paths provided without quotes are preprocessed by the shell and any wildcard or other patterns cannot be prevented.Context variablesUseful context variables are added before any other variable is loaded. Some are global for all templates rendered, and some are template-specific.VariableScopeDescription__python_version__GlobalPython version__jinja2_version__GlobalJinja2 version__j2gpp_version__GlobalJ2GPP version__user__GlobalName of the current user__pid__GlobalProcess ID of the current process__ppid__GlobalProcess ID of the parent process__working_directory__GlobalWorking directory__output_directory__GlobalOutput directory__date__GlobalDate in the formatDD-MM-YYYY__date_inv__GlobalDate in the formatYYYY-MM-DD__time__GlobalTime in the formathh:mm:ss__datetime__GlobalTimestamp in the formatYYYY-MM-DD hh:mm:ss__source_path__TemplatePath of the source template file__output_path__TemplatePath where the template is renderedBuilt-in filtersIn addition to theJinja2 built-in filters, J2GPP also defines many useful filter functions.All functions from the Python librariesmathandstatisticsare made available as filters. This includes useful functions such assqrt,pow,log,sin,cos,floor,ceil,mean,median,variance,stdev, ...An operation can be applied to all elements of a list with the filterslist_add,list_sub,list_mult,list_div,list_mod,list_remandlist_exprespectively for the Python operators+,-,*,/,%,//and**.Text alignment can be controlled with the Python functionsljust,rjustandcenter.Case and formatting can be controlled with the Python functionstitleandswapcase, or the added functionscamel,pascal,snakeandkebab.camelandpascalwill remove the underscores and hyphens by default but leave the dots ; that behaviour can be changed by providing the filter argumentsremove_underscore,remove_hyphenandremove_dotasTrueorFalse.snakeandkebabwill group capitalized letters and preserve capitalization by default ; that behaviour can be changed by providing the filter argumentspreserve_capsandgroup_capsasTrueorFalse.Paragraph formatting is facilitated by multiple filters that should be used on afilterblock.reindentremoves pre-existing indentation and sets new one.autoindentremoves pre-existing indentation and sets new indent by incrementing and decrementing the depth based on start and end delimiters of blocks provided by thestartsandendslists of strings provided by argument (culry braces by default).alignaligns every line of the paragraph by columns, left before§, right before§§.When using a list of dictionaries with a key in common, you can get the list element with the minimum or maximum value of that attribute using the filtersel_of_max_attrorel_of_min_attr.When using two-level dictionaries, the key corresponding to the minimum or maximum with regards to a specified attribute using the filterskey_of_max_attrorkey_of_min_attr.You can count the number of occurrences of a value in a list using thecountfilter.Thewriteandappendfilters can be used to export the content of a filter to another file whose path is provided as argument to the filter. The path can be absolute or relative to the output rendered base template. By default, the content of the filter is not written to the base rendered template ; this behaviour can be changed by providing the filter argumentpreserveasTrue. The source template can also be prevented from resulting in a generated file by providing the filter argumentwrite_sourceasFalse, and only the content ofwriteandappendblocks will generate files.Thewarninganderrorfilters can be used to throw warnings and errors from the template that will be displayed in the J2GPP logs. The filter is applied to a block, replaces the block with nothing and throws the warning or error with the content of the block as comment. The filter works with conditional blocks if the version of Jinja2 installed supports the@render_time_onlydecorator.Process directoriesWhen the source path provided corresponds to a directory, J2GPP will look for any template files in the source directory tree. If no output directory argument is provided, the rendered files will be written next to the source templates. If an output directory is provided, the source directory tree structure will be copied to the output directory with only the rendered files.For instance, suppose we have the following directory structure :.
└── test_dir
    ├── sub_dir_1
    │   ├── deep_dir
    │   │   └── template_1.txt.j2
    │   └── non_template_1.txt
    ├── sub_dir_2
    │   └── non_template_2.txt
    └── template_2.txt.j2When we execute the commandj2gpp ./test_dir/, we will get :.
└── test_dir
    ├── sub_dir_1
    │   ├── deep_dir
    │   │   ├── template_1.txt
    │   │   └── template_1.txt.j2
    │   └── non_template_1.txt
    ├── sub_dir_2
    │   └── non_template_2.txt
    ├── template_2.txt
    └── template_2.txt.j2But if we provide an output directory with the commandj2gpp ./test_dir/ --outdir ./out_dir/, we will get :.
├── test_dir
│   ├── sub_dir_1
│   │   ├── deep_dir
│   │   │   └── template_1.txt.j2
│   │   └── non_template_1.txt
│   ├── sub_dir_2
│   │   └── non_template_2.txt
│   └── template_2.txt.j2
└── out_dir
    ├── sub_dir_1
    │   └── deep_dir
    │       └── template_1.txt
    └── template_2.txtWe can also tell J2GPP to copy the non-template files with the commandj2gpp ./test_dir/ --outdir ./out_dir/ --copy-non-template, then we will get :.
├── test_dir
│   ├── sub_dir_1
│   │   ├── deep_dir
│   │   │   └── template_1.txt.j2
│   │   └── non_template_1.txt
│   ├── sub_dir_2
│   │   └── non_template_2.txt
│   └── template_2.txt.j2
└── out_dir
    ├── sub_dir_1
    │   ├── deep_dir
    │   │   └── template_1.txt
    │   └── non_template_1.txt
    ├── sub_dir_2
    │   └── non_template_2.txt
    └── template_2.txtOr even to process non-templates files as templates anyway with the commandj2gpp ./test_dir/ --outdir ./out_dir/ --render-non-template, then we will get :.
├── test_dir
│   ├── sub_dir_1
│   │   ├── deep_dir
│   │   │   └── template_1.txt.j2
│   │   └── non_template_1.txt
│   ├── sub_dir_2
│   │   └── non_template_2.txt
│   └── template_2.txt.j2
└── out_dir
    ├── sub_dir_1
    │   ├── deep_dir
    │   │   └── template_1.txt
    │   └── non_template_1_j2gpp.txt
    ├── sub_dir_2
    │   └── non_template_2_j2gpp.txt
    └── template_2.txtSupported formats for variablesJinja2 supports variables types from python. The main types are None, Boolean, Integer, Float, String, Tuple, List and Dictionary. J2GPP provides many ways to set variables and not all types are supported by each format.Command line defineDefines passed by the command line are interpreted by the Pythonast.literal_eval()function which supports Python syntax and some additional types such asset().j2gpp./foo.c.j2--definetest_none=None\test_bool=True\test_int=42\test_float=3.141592\test_string1=lorem\test_string2="lorem ipsum"\test_tuple="(1,2,3)"\test_list="[1,2,3]"\test_dict="{'key1': value1, 'key2': value2}"\test_dict.key3=value3YAMLtest_none1:
test_none2: null

test_bool1: true
test_bool2: false

test_int: 42
test_float: 3.141592

test_string1: lorem ipsum
test_string2:
  single
  line
  text
test_string3: |
  multi
  line
  text

test_list1: [1,2,3]
test_list2:
  - 1
  - 2
  - 3

test_dict:
  key1: value1
  key2: value2
  key3: value3JSON{"test_none":null,"test_bool1":true,"test_bool2":false,"test_int":42,"test_float":3.141592,"test_string":"lorem ipsum","test_list":[1,2,3],"test_dict":{"key1":"value1","key2":"value2","key3":"value3"}}XMLNote that XML expects a single root element. To avoid having to specify the root element when using the variables in a template, J2GPP automatically removes the root element level if it is named "_".<_><test_none></test_none><test_bool1>true</test_bool1><test_bool2>false</test_bool2><test_int>42</test_int><test_float>3.141592</test_float><test_string>loremipsum</test_string><test_list>1</test_list><test_list>2</test_list><test_list>3</test_list><test_dict><key1>value1</key1><key2>value2</key2><key3>value3</key3></test_dict></_>TOMLtest_bool1=truetest_bool2=falsetest_int=42test_float=3.141592test_string1="lorem ipsum"test_string2="""multilinetext"""test_list=[1,2,3][test_dict]key1="value1"key2="value2"key3="value3"INI/CFGNote that INI file expects data to be divided in sections with a header in square brackets. To avoid having to specify the root element when using the variables in a template, J2GPP automatically flattens the section whose header is "_".[_]test_bool1=Truetest_bool2=Falsetest_int=42test_float=3.141592test_string="lorem ipsum"test_list=[1,2,3][test_dict]key1=value1key2=value2key3=value3ENVtest_bool1 = True
test_bool2 = False

test_int = 42
test_float = 3.141592

test_string = lorem ipsum

test_list = [1,2,3]

test_dict = {'key1':'value1','key2':'value2','key3':'value3'}CSV/TSVCSV and TSV are interpreted as a list of objects with the same attributes. They are converted to a list of dictionaries whose name is the first cell of each line and the keys are the headers of each column.CSV and TSV use the same loader, just with different delimiters. A different delimiter can be provided with the argument--csv-delimiter. To use the delimiter in a value, it can be escaped by defining an escape character with the argument--csv-escapechar, for instance the backslash "\".By default, the whitespace around the keys and values in the CSV is stripped. This behaviour can be disabled with the argument--csv-dontstrip.keys,key1,key2,key3
test_dict1,1,2,3
test_dict2,11,12,13
test_dict3,21,22,23keys  key1  key2  key3
test_dict1  1  2  3
test_dict2  11  12  13
test_dict3  21  22  23Scripting in J2GPP templatesAn advanced use of the template feature and filters of Jinja2 and J2GPP allow this tool to do some amount of scripting. The main features allowing this usage are explained in this section.Conditional and do extensionThe basic Jinja2 feature of conditional blocksif/elif/elsecan be used alongside thedoextension to more easily manipulate variables and complex objects such as Python dictionaries.Note that, if possible, it is preferable to process the variables after loading and before rendering by providing a Python function with the--vars-post-processorcommand line argument.Conditional and filtersAs filters are Python functions, they can be very powerful, especially when coupled with the use of conditional blocks. However, Jinja2 optimizes processing by running some filters during the compilation phase, while conditional blocks are resolved at render time.To fix this, you can use the@render_time_onlydecorator to force a filter or test to execute at render time only. This decorator iscurrentlyonly available by installing a custom fork of Jinja2 :gitclonehttps://github.com/Louis-DR/jinja.gitcdjinja
pip3install./Throw errors and warnings from templateIf the version of Jinja2 installed supports the@render_time_onlydecorator, then thewarninganderrorfilters allow to throw warnings and erros from the template and display them in the J2GPP logs. This is useful with conditionals to perform data sanity checks for instance.Writing filesThe J2GPP filterswriteandappendallow exporting the content of a block to another file. This can be used for a file combining elements contributed by multiple templates, for alternative versions of a file from a single template, for generating small annex files to a large template, for generating a files for each element in a list, etc. When coupled with theincludeormacrostatements with nested temlates, it allows for even more complex outputs.Note that if the installed Jinja2 version doesn't support the@render_time_onlydecorator, using thewriteandappendfilters in conditional blocks may results in unwnated behaviour.Write exampleFor example, with the parent templateparent.txt.j2:This will be rendered to the parent output file
{% filter write("child.txt") %}
This will be rendered to the child output file
{% endfilter %}>>>tree
.
└──parent.txt.j2

>>>j2gpp./parent.txt.j2[...]>>>tree
.
├──child.txt
├──parent.txt
└──parent.txt.j2

>>>catparent.txt
Thiswillberenderedtotheparentoutputfile

>>>catchild.txt
ThiswillberenderedtothechildoutputfileAppend exampleIf the file doesn't exists, theappendfilter will create it and be equivalent towrite. However, if the file already exists,writewill override it whileappendwill append to the end of it. For example, with the parent templateparent.txt.j2:This will be rendered to the parent output file
{% filter append("child.txt") %}
This will be rendered to the child output file
{% endfilter %}>>>tree
.
└──parent.txt.j2

>>>j2gpp./parent.txt.j2[...]>>>tree
.
├──child.txt
├──parent.txt
└──parent.txt.j2

>>>catparent.txt
Thiswillberenderedtotheparentoutputfile

>>>catchild.txt
Thiswillberenderedtothechildoutputfile

>>>j2gpp./parent.txt.j2[...]>>>catchild.txt
Thiswillberenderedtothechildoutputfile
Thiswillberenderedtothechildoutputfile

>>>j2gpp./parent.txt.j2[...]>>>catchild.txt
Thiswillberenderedtothechildoutputfile
Thiswillberenderedtothechildoutputfile
ThiswillberenderedtothechildoutputfilePath argumentThewriteandappendfilters require at least one argument, the name or path of the second file to generate. The path can be absolute, or relative to the generated file of the parent template. Non-existing directories will be created. For example, with the parent templateparent.txt.j2:{% filter write("foo/child.txt") %}
This will be rendered to the child output file
{% endfilter %}>>>tree
.
└──src└──parent.txt.j2

>>>j2gpp./src/parent.txt.j2--outdir./gen/[...]>>>tree
.
├──src
│└──parent.txt.j2
└──gen├──foo│└──child.txt└──parent.txtWriting to both filesBy default, the content of the block is only written to the child file, and is not written to the parent rendered template. This behaviour can be changed by providing the filter argumentpreserveasTrue. For example, with the parent templateparent.txt.j2:{% filter write("child.txt", preserve=True) %}
This will be rendered to both parent and child output files
{% endfilter %}>>>j2gpp./parent.txt.j2--outdir./gen/[...]>>>cat./parent.txt
Thiswillberenderedtobothparentandchildoutputfiles

>>>cat./child.txt
ThiswillberenderedtobothparentandchildoutputfilesSkipping the parent fileThe source template can also be prevented from resulting in a generated file by providing the filter argumentwrite_sourceasFalse, and only the content ofwriteandappendblocks will generate files. For example, with the parent templateparent.txt.j2:This will not be written to any file
{% filter write("child.txt") %}
This will be rendered to the child output file
{% endfilter %}>>>tree
.
└──parent.txt.j2

>>>j2gpp./parent.txt.j2[...]>>>tree
.
├──child.txt
└──parent.txt.j2

>>>catchild.txt
Thiswillberenderedtothechildoutputfile