npm package: @ndcode/jst
Description: JavaScript Templates for HTML
Latest version: 0.1.5

# JavaScript Template system

An NDCODE project.

## Overview

The `jst` package exports a single function
  `jst(text, options)`
which parses the given text in a JavaScript dialect which can contain HTML-like
constructs, then generates equivalent plain JavaScript code which when run,
would generate the requested HTML. As well as the HTML generation, there can be
normal JavaScript logic to influence the generated page or substitute into it.

The system is inspired by Pug (formerly Jade) templates, but emphasizing the
JavaScript, for instance you do not need a `-` sign at the start of each line,
and the parser automatically follows the JavaScript block scope. It is also
inspired by JSX templates, but the embedded HTML is less verbose, there are no
closing tags since HTML is bracketed by `{ }` rather than by `<tag>...</tag>`.

## Template syntax

### HTML tags in templates

The plain JavaScript in the file is absolutely normal and is expected to use
CommonJS conventions, for instance, `require(...)` is supported and so forth.

The embedded HTML uses a syntax similar to JavaScript function definitions,
  `function(arg1, arg2, ...) {...}`
except that instead of the word "function" any valid HTML (or other) tag name
is allowed, and instead of just argument names, HTML attributes of the form
  `attrname`
or
  `attrname=value`
are supported. In the `attrname=value` syntax, `value` is any valid JavaScript
expression. No commas are required between attributes, so the parser has to
automatically detect where one expression ends and another begins (similarly to
how automatic semicolon insertion works in regular JavaScript).

The `( )` around the attributes is not necessary when there are no attributes.
Thus a simple HTML file could be written as the following JavaScript template:

```
html(lang="en") {
  head {}
  body {}
}
```
and this would translate to the HTML file:
```htm
<html lang="en"><head></head><body></body></html>
```

For certain particular HTML tags such as `img`, no closing tag is generated in
the output HTML. However, the empty `{ }` must still be given in the template.
This is partly for the parser's convenience, since it depends on recognizing
the `{ }` to distinguish between an HTML template and ordinary JavaScript code.
It is also more uniform: there is no need to remember which tags are special.

### Regular text in templates

Regular text is placed in the generated HTML by simply quoting it. That is, if
a statement is seen consisting of only a JavaScript string or template string,
the string is converted to HTML by escaping it (helping to guard against any
HTML injection attacks), and then output as part of some enclosing template.

For example:
```js
html(lang="en") {
  body {'Hello, world'}
}
```

This generates the text:
```html
<html lang="en"><body>Hello, world</body></html>
```

If the text is multi-line, backquoted (template) strings are used, to allow
embedded newlines. When the entire text is held in a variable, e.g. `myvar`, a
template string such as ```${myvar}``` should be used, to convert it into a
standalone statement consisting of only a string constant, and hence into HTML.

Note that ordinary single- or double-quoted string constants should be used in
preference to template strings where possible, since the constant strings will
be HTML-escaped at template compile time rather than at template run-time.

Note that in ordinary HTML, certain tags are more sensitive to whitespace than
others, according to complicated rules about inline block elements and so on.
This is never an issue with JavaScript templates, we can use as much indenting
and other whitespace as we want, and only quoted whitespace will be output.

### Special tags in templates

The HTML `script` tag is treated specially, because it contains JavaScript,
which is understood directly by the template parser. The JavaScript inside the
script tag is minified (comments stripped, unnecessary braces and parentheses
removed, semicolons added and so on), and then converted to a string constant,
which will be copied out to the HTML file whenever the template is executed.

The HTML `style` tag is treated specially, because it contains CSS, which we
handle by switching temporarily to another parser that understands CSS. We use
a slightly modified version of the `clean-css` package to do this. The result
is then minified and converted to a string constant, similar to `script` tags.

Note that the `script` and `style` tags cannot contain substitutions, in the
way that ordinary JST code can. This is because (i) the entire contents of the
special tag are minified and stringified at template compile time, so the
content of the special tag is the same each time the page is generated, and
(ii) it runs elsewhere, which doesn't have access to the template variables.

### HTML class and id shorthand

The tag name can be followed by `#name` as shorthand for the HTML attribute
  `id="name"`
or by `.name` as shorthand for the HTML attribute
  `class="name"`
and these can be repeated as needed, the `id` attribute collects all `#name`
separated by spaces and the `class` attribute collects all `.name` similarly.
These must come before any ordinary attributes (before an opening parenthesis).

### Parser limitations

Certain tag or attribute names present difficulty since they contain `-` signs
or other characters invalid in JavaScript identifiers, or they may be reserved
words in JavaScript. The parser copes with this quite well (most syntax errors
can already be re-parsed as tag or attribute names), but in difficult cases it
could be necessary to quote the tag and/or attribute names. For example,
  `div.class-1.class-2 {}`
doesn't compile because "1." is a floating point number, for now we write it
  `div.'class-1'.class-2 {}`
although we expect that this restriction can be lifted in a future version.

Also, the `style` parser for CSS code can be confused by tag, `id` or `class`
names that end with `-style`. So we should be careful of code like this,
  `div.my-style {...}`
since the `div.my-` prefix won't be seen until parsing is complete, hence the
parser will switch to CSS parsing inside the braces. Therefore, quote it like
  `div.'my-style' {...}`
although again, we expect that this restriction can be lifted in the future.

Another slight limitation of the current parser is that it is more permissive
than normal in parsing regular JavaScript code, for instance commas are not
required between function arguments, because an HTML template is basically
parsed as a function call until the opening `{` is seen to identify it as a
template. This can also likely be improved in a future version of the system.

## Expression vs statement templates

HTML templates occuring at expression level will be completely rendered to a
string, and the resulting string returned as the value of the expression.

HTML templates occurring at statement level are treated somewhat like quoted
strings, in that the generated HTML will become part of an enclosing template.

Here is a complete example showing template substitution and HTML expressions:
```js
let lang = 'en'
let name = 'John'
console.log(
  html(lang=lang) {
    body {`Hello, ${name}`}
  }
)
```

Running the above program will generate the output:
```html
<html lang="en"><body>Hello, John</body></html>
```

## Template output buffer

It is also possible to use statement-level HTML templates or strings to build
up an output buffer, which can be used for whatever purpose. The output buffer
must be called `_out` and must be a JavaScript array of string fragments. As
each output fragment (such as an opening or closing tag or an embedded string)
is generated, it will be sent to the output buffer by an `_out.push(...)` call.

If there is dynamically generated text in the template, then it will be esacped
by the sequence `.replace("&", "&amp;").replace("<", "&lt;")`, this is chosen
so that no external dependency such as the `html-entities` package is needed,
and so that unnecessary escaping of characters such as `'` or `>` is not done.
Attributes are done similarly, except that `"` is replaced instead of `<`. We
assume a `UTF-8` environment, so there is really little need for HTML entities.

For example, consider a realistic template which we use on our demo server:
```js
html(lang=lang) {
  head {
    link(rel="stylesheet" type="text/css" href="css/styles.css") {}
  }
  body {
    p {`Hello, ${name}`}
  }
}
```

This compiles to the following plain JavaScript code:
```js
_out.push("<html lang=\"" + lang.toString().replace("&", "&amp;").replace("\"", "&quot;") + "\">");
{
  _out.push("<head>");
  _out.push("<link rel=\"stylesheet\" type=\"text/css\" href=\"css/styles.css\">");
  _out.push("</head>");
}
{
  _out.push("<body>");
  {
    _out.push("<p>");
    _out.push(`Hello, ${name}`.replace("&", "&amp;").replace("<", "&lt;"));
    _out.push("</p>");
  }
  _out.push("</body>");
}
_out.push("</html>");
```

If invoked as an expression, the same code will be generated, but wrapped in an
Immediately-Invoked Function Expression (IIFE) that creates the `_out` buffer,
executes the above, then returns the buffer concatenated into a single string.

## Usage example

Suppose we want to try out the JST template processor. Returning to the example
from the section "Expression vs Statement templates", we could create a file
called `mytemplate.jst` containing the example JST code to print a simple page:
```js
let lang = 'en'
let name = 'John'
console.log(
  html(lang=lang) {
    body {`Hello, ${name}`}
  }
)
```js
We could then create an example JavaScript program `example.js` to read,
compile and then execute the template, as follows:
```js
let fs = require('fs')
let jst = require('@ndcode/jst')

eval(jst(fs.readFileSync('mytemplate.jst')))
```
To run this, we would have to run the `node` interpreter at the command line:
```sh
node example.js
```
This produces the output described earlier:
```html
<html lang="en"><body>Hello, John</body></html>
```

## Command line preprocessor

Using the programmatic interface is overkill in many situations, e.g. if we
just want to see the generated JavaScript code, or if we want to compile all
pages of our website to JavaScript ahead of time by a `Gruntfile` or similar.

Hence, we provide the command line interface `jst` as a convenient way to
convert JST to JavaScript. Using the command-line, the above example becomes:
```
jst <mytemplate.jst >mytemplate.js
node mytemplate.js
```
Inspecting the JavaScript file `mytemplate.js` shows how the HTML is generated:
```
let lang = 'en';
let name = 'John';
console.log((() => {
  let _out = [];
  _out.push("<html lang=\"" + lang.toString().replace("&", "&amp;").replace("\"", "&quot;") + "\">");
  {
    _out.push("<body>");
    _out.push(`Hello, ${name}`.replace("&", "&amp;").replace("<", "&lt;"));
    _out.push("</body>");
  }
  _out.push("</html>");
  return _out.join('');
})());
```

Various command line options are available for setting indentation and similar.
There is also a `--wrap` option which adds some prologue and epilogue code,
which is part of a larger system we have designed, for allowing templates to
embed each other and so on. See the `jst_server` and `jst_cache` modules for
more information. If the extra complexity isn't needed, simply omit `--wrap`.

## JSTizer program

In development, we often want to refer to example code from the Web, e.g. if we
want to incorporate the Bootstrap front-end framework or if we want to create a
web-form or some other common web-development task. The example code is usually
written in HTML. We can drop this straight into a JST project by JSTizing it.

See the `jstize` module for more information. Essentially the workflow is (i)
get some HTML in a file `*.html`, it does not need to be a complete page, but
must be a syntactically valid and complete HTML fragment, (ii) use the JSTizer
to convert this into an equivalent `*.jst` file, (iii) use the JST system, by
either the programmatic API or the command-line tool, to convert this into an
equivalent `*.js` file, (iv) run the `*.js` file to recreate the original HTML.
Of course, the `*.jst` template can be modified to add any needed extra logic.

## GIT repository

The development version can be cloned, downloaded, or browsed with `gitweb` at:
https://git.ndcode.org/public/jst.git

## License

All of our NPM packages are MIT licensed, please see LICENSE in the repository.
We also gratefully acknowledge the `acorn` team for the original MIT license of
their JavaScript parser, which we've heavily modified to create the JST parser.

## Contributions

The `jst` system is under active development (and is part of a larger project
that is also under development) and thus the API is tentative. Please go ahead
and incorporate the system into your project, or try out our example webserver
built on the system, subject to the caution that the API could change. Please
send us your experience and feedback, and let us know of improvements you make.

Contact: Nick Downing <nick@ndcode.org>