npm package: @ndcode/jst_cache
Description: JavaScript Templates for HTML, caching front-end
Latest version: 0.1.0

# Build Cache wrapper for JavaScript Template system

An NDCODE project.

## Overview

The `jst_cache` package exports a single constructor
  `JSTCache(root, args, diag)`
which must be called with the `new` operator. The resulting cache object stores
compiled JavaScript templates. Each template is, in general, a function that
can be called with certain arguments, and which will generate the HTML text for
a particular page or part of a page. The arguments can vary with each template.

The `JSTCache` object takes care of loading template source code, compiling it
using the `jst` package, and then executing the resulting JavaScript code. This
initial execution is intended to allow the template to load any resources it
needs, and generally set things up so that it is ready to generate HTML. The
initial execution returns the object which is stored in the cache. As mentioned
this is normally a function, but could be something else, e.g. any JSON object.

See the `build_cache`, `disk_build` and `jst` packages for more information.
The `JSTCache` object is essentially a wrapper object which routes the request 
between these packages, to ensure that the compiled template is retrieved from
either RAM or disk if available, or is compiled and stored to RAM and to disk.

As well as wrapping the `build_cache`, `disk_build` and `jst` functionality
into a convenient "point and shoot" interface for template loading, `jst_cache`
also provides a dependency resolution service for the compiled template. If the
compiled template wants to load further templates during initialization or run-
time, it calls back into `jst_cache`, which recursively loads the dependency.

## Template examples

See the `jst` package for more information about what can be in a template.
Essentially, the templates are written in a dialect of JavaScript which allows
HTML constructs to be specified directly, returning a string containing HTML.

HTML templates occuring at expression level will be completely rendered to a
string, and the resulting string returned as the value of the expression.

HTML templates or strings (either single or double quoted strings, or template
strings) occurring at statement level, become part of an enclosing template.

For example, the JavaScript code `let text = p {'Hello, world'}` sets `text`
to the value `<p>Hello, world</p>`. This consists of an HTML template `p { }`
occurring at expression level, then the string `'Hello, world'` at statement
level inside the `p { }` template (any valid JavaScript statement is allowed).

Here is a complete example showing template substitution and HTML expressions:
```
let lang = 'en'
let name = 'John'
console.log(
  html(lang=lang) {
    body {`Hello, ${name}`}
  }
}
```

Running the above program will generate the output:
```
<html lang="en"><body>Hello, John</body></html>
```

## Template output buffer

It is also possible to use statement-level HTML templates or strings to build
up an output buffer, which can be used for whatever purpose. The output buffer
must be called `_out` and must be a JavaScript array of string fragments. As
each output fragment (such as an opening or closing tag or an embedded string)
is generated, it will be sent to the output buffer by an `_out.push(...)` call.

If there is dynamically generated text in the template, then it will be esacped
by the sequence `.replace("&", "&amp;").replace("<", "&lt;")`, this is chosen
so that no external dependency such as the `html-entities` package is needed,
and so that unnecessary escaping of characters such as `'` or `>` is not done.
Attributes are done similarly, except that `"` is replaced instead of `<`. We
assume a `UTF-8` environment, so there is really little need for HTML entities.

For example, consider a realistic template which we use on our demo server:
```
html(lang=lang) {
  head {
    link(rel="stylesheet" type="text/css" href="css/styles.css") {}
  }
  body {
    p {`Hello, ${name}`}
  }
}
```

This compiles to the following plain JavaScript code:
```
_out.push("<html lang=\"" + lang.toString().replace("&", "&amp;").replace("\"", "&quot;") + "\">");
{
  _out.push("<head>");
  _out.push("<link rel=\"stylesheet\" type=\"text/css\" href=\"css/styles.css\">");
  _out.push("</head>");
}
{
  _out.push("<body>");
  {
    _out.push("<p>");
    _out.push(`Hello, ${name}`.replace("&", "&amp;").replace("<", "&lt;"));
    _out.push("</p>");
  }
  _out.push("</body>");
}
_out.push("</html>");
```

If invoked as an expression, the same code will be generated, but wrapped in an
Immediately-Invoked Function Expression (IIFE) that creates the `_out` buffer,
executes the above, then returns the buffer concatenated into a single string.

## Template dependencies

Templates may depend on other templates. Templates are imported imported using
the syntax `_require(...)`, similarly to the syntax `require(...)` in CommonJS.

We use `_require()` for templates instead of the normal `require()` because:

(1) We do not want to interfere with other symbols in the namespace, you can
also use `require()` in your templates to import ordinary CommonJS modules;

(2) Absolute paths have a different meaning in `_require()` versus `require()`,
for `_require()` they will be taken relative to the webserver's document root;

(3) Using `_require()` says that the file is in `*.jst` format, and hence it
should be parsed and any HTML constructs converted into regular JavaScript; and

(4) The template importing is an asynchronous operation, so you normally use
`await _require(...)`, whereas `require()` is synchronous. Because templates
which have been edited are recompiled automatically by the running webserver,
we don't want to stop the world, especially as templates are slow to compile.

A slight difference between CommonJS exports and JavaScript template exports,
is that where CommonJS modules set the dedicated symbol `module.exports` to
whatever is exported, JavaScript templates instead `return` what is exported.

## Template paths and variables

Similarly to CommonJS, the variable `_require` is injected by `jst_cache` and
is available to all templates. Other variables injected by `jst_cache` are
`_pathname` which is the path to the currently executing template, and `_root`
which is the root directory for absolute pathname references in `_require()`.

Thus, paths sent to `_require()` are resolved relative to either `_root` if
absolute (beginning with `/`) or the directory part of `_pathname` if relative
(not beginning with `/`). Each template gets its own `_require()` function,
which knows the `_pathname` and `_root` values of the calling template.

If further variables should be made available to the template, they can be
included as a dictionary in the `args` parameter to `JSTCache()`. For example,
```js
let jst_cache = new JSTCache(
  '/home/myname/www',
  {myvar: 'some value'}
)
let page = jst_cache.get('/home/myname/www/blog/page.jst')
```
parses and executes the template `/home/myname/www/blog/page.jst`, with the
variables `_pathname` set to `'/home/myname/www/blog/page.jst'`, `_root` set to
`'/home/myname/www'`, and `myvar` set to `'some value'`. Suppose `page.jst` is:
```js
let footer = _require('/site/footer.jst')
```
or
```js
let footer = _require('../site/footer.jst')
```
then the template `/home/myname/www/site/footer.jst` is parsed and executed,
with `_pathname` set to `'/home/myname/www/site/footer.jst'`, and the other
injected variables `_root` and `myvar` set from the original `JSTCache()` call.

## Multi-part template examples

### Templating with HTML text strings

Here is an example of a template which takes the body of a page (as text which
may contain HTML tags), and wraps it in the standard `html` and `body` tags to
create a complete page. The idea is to use this to generate every page, so that
additional commands, such as viewport commands or script tags, can be set here.

Save this as `page.jst`, since it's the template for all pages in the system:
```
return body_text => html {
  head {
    link(rel="stylesheet" type="text/css" href="css/styles.css") {}
  }
  body {
    _out(body_text)
  }
}
```
Save this as `index.html.jst`, as it's the template to generate `index.html`:
```
let page = await _require('/page.jst')

return page(p {'Hello, world'})
```
Note that here we have somewhat hacked into the internals of the templating
system by directly appending the body text onto the output buffer `_out`. It is
intentional that you can access the internals for efficiency reasons, and that
you can, for example, avoid HTML escaping of particular text where appropriate.

### Templating with callbacks

Another way, which is more sophisticated, is to use a callback function to
generate the body. If doing this, we pass the output buffer `_out` into the
callback function, so that the entire HTML of the page can be generated in an
unbroken stream, without having to concatenate the partial page repeatedly.

The page wrapper template `page.jst` which takes a body callback and calls it:
```
return body_callback => html {
  head {
    link(rel="stylesheet" type="text/css" href="css/styles.css") {}
  }
  body {
    body_callback(_out)
  }
}
```

Then the specific page template `index.html.jst` which provides the callback:
```
let page = await _require('/page.jst')

return html(
  _out => {
    p {'Hello, world'}
  }
)
```

Note that in the above example, the `{ }` around the callback function body is
essential, unlike in regular JavaScript, so that `p {...}` occurs in statement
rather than expression context. As an expression it would return a string which
would be ignored, since the page template is expecting output to be in `_out`.

## File management

The `JSTCache` object a "point and shoot" interface which makes it very easy to
manage on-disk templates. You just call `JSTCache.get()` stating a pathname to
a template, normally a `*.jst` file, and the template is parsed, compiled to
JavaScript, and evaluated, with the evaluation result cached for future reuse.

What the `JSTCache.get()` function returns depends on the template source code.
Template exports are similar but slightly different to CommonJS module exports,
as noted in examples above. Usually, `JSTCache.get()` returns a JavaScript
function, which you call to generate HTML each time the page is to be served.

The HTML-generating function is re-useable in this way for efficiency reasons,
since after the initial compilation, the cached version can be reused each time
the page is served, but called with different arguments, resulting in different
substitutions made on the page. For example, consider the following template:
```
return (lang, name) => html(lang=lang) {
  body {
    p {`Hello, ${name}`}
  }
}
```
Suppose the above is saved as `/hello/index.html.jst` under your document root
`/var/www/html`. When you instantiate the `JSTCache` object giving the document
root, and then call the `get(pathname)` instance function, you receive a
function of two arguments `lang` and `name`, which returns the HTML string.

Note that `_require()` paths beginning with `/` are taken relative to the
document root passed into the `JSTCache()` constructor call, whereas paths
beginning with anything else are taken relative to the `dirname` extracted from
the current value of `_pathname` as seen by the template. Each template has its
own `_pathname` and thus its `_require()` calls are relative to its own source.

## Memory vs disk caching of templates

Templates compiled using `JSTCache.get()` are cached in memory, as long as the
same `node` interpreter is running, so that they can be retrieved using either
`JSTCache.get()`, or equivalently `_require()` inside a template, and they will
not be re-executed. In the above example, if you call `JSTCache.get()` twice,
you get the same object twice (it is a 2-argument function returning `String`).

As well as this, the compiled templates are also cached on disk, which requires
the document root to be writeable. For instance if you compile `index.html.jst`
in a `hello/` subdirectory of the document root, you get a hidden file with the
same name prefixed by `.` in the same directory, here `hello/.index.html.jst`.

The main reason for the disk caching is really to fool the node.js `require()`
system into using correct relative paths, if the template uses `require()` as
well as `_require()`. However, it is also handy that the templates only need to
be re-evaluated and not recompiled if the webserver is stopped and restarted.

Before using either a disk-cached or a memory-cached template, the modification
times are checked and the template is recompiled if it is stale. Thus, you can
edit your website while it is live, and each page will be recompiled as needed
just before serving. Templates deleted on disk are not returned from the cache.

Note: If the document root is not writeable, a simple expedient is to make sure
that all `.*.jst` files are up to date, so no write is attempted. You could do
this by firing up a development instance of the server in a writeable directory
and then indexing the whole site, for example by a recursive `wget` invocation.

## To be implemented

It is intended that we will shortly add a timer function (or possibly just a
function that the user should call periodically) to flush built templates from
the cache after a stale time, on the assumption that the template might not be
accessible or wanted anymore. For example, if the templates are HTML pages, the
link structure of the site may have changed to make some pages inaccessible.

## GIT repository

The development version can be cloned, downloaded, or browsed with `gitweb` at:
https://git.ndcode.org/public/jst_cache.git

## License

All of our NPM packages are MIT licensed, please see LICENSE in the repository.

## Contributions

The `jst` system is under active development (and is part of a larger project
that is also under development) and thus the API is tentative. Please go ahead
and incorporate the system into your project, or try out our example webserver
built on the system, subject to the caution that the API could change. Please
send us your experience and feedback, and let us know of improvements you make.

Contact: Nick Downing <nick@ndcode.org>