Repository: mozilla/nunjucks (js)

Issue:
Provide a way to discover a template's dependencies
nunjucks doesn't provide a way to discover a template's direct or transitive dependencies i.e. the child and grandchild templates that are loaded (via `extends`, `import` or `include`) when a parent template is rendered. This is needed to [integrate](https://github.com/devmattrick/parcel-plugin-nunjucks/issues/1) ([efficiently](https://github.com/devmattrick/parcel-plugin-nunjucks/pull/3)) with bundlers such as [Parcel](https://github.com/parcel-bundler/parcel), which track changes in dependencies and use them to trigger rebuilds.

(Note: this is orthogonal to nunjucks' caching support via chokidar, though the implementations may overlap.)

Ideally, this would be scoped to a particular `Environment#render` call, but there doesn't appear to be any space in the `render` method's parameters to squeeze in additional options, or to return an additional value. Maybe a sister method with a more flexible interface?

```javascript
// result is an object rather than a string
const result = env.parse(templatePath, { context: ..., trackDependencies: true })
console.log('dependencies:', result.dependencies)
console.log('source:', result.source)
```

This makes dependency tracking optional, to err on the side of efficiency, but they could be included by default as it's a new API, and the overhead is unlikely to be high e.g.:

```javascript
const result = env.parse(templatePath, { context: ... })
```

Another way to do it is via an EventEmitter API (which is already used, lightly, in loaders) e.g.:

```javascript
env.on('dependency', ({ name, path, parent }) => { ... })
```

This has several advantages:

- it doesn't clutter or crowd any existing APIs
- it's easier to implement
- it's easy to extend
- if there's a performance concern, it could be toggled on/off with a constructor option e.g. `new Environment(loaders, { trackDependencies: true })`

The main disadvantage is that it doesn't *directly* answer the question "What are this template's dependencies?" i.e. it requires extra work to scope the results to a particular template/render. (This also implies the use of a full EventEmitter implementation rather than the cut-down version that's currently used for loaders, and suggests the addition of some new events e.g. `render:start` and `render:end`.)

---

I'm envisaging each dependency as an object with the following core fields:

```javascript
{
    name: "../macros/util.html.njk",
    path: "/home/foo/dev/example/src/html/macros/util.html.njk",
    parent: "/home/foo/dev/example/src/html/screens/layout.html.njk",
}
```

Optional fields could include `cached` (a boolean indicating that the template was retrieved from its loader's cache) and maybe `async` (the template was retrieved by an async loader).

# Example

### layout.html

```xml
{% include "../components/header.html" %}
<h1>Body</h1>
{% include "../components/footer.html" %}
```

### header.html

```xml
<h1>Header</h1>
```

### footer.html

```xml
<h1>Footer</h1>
{% include "./copyright.html" %}
```

### copyright.html

```xml
Copyright ⓒ example.com 2018
```

### render

```javascript
    const result = env.parse("src/html/screens/layout.html")
    console.log(result.dependencies)
```

### result

```javascript
[
    {
        name: "src/html/screens/layout.html",
        path: "/home/foo/example/src/html/screens/layout.html",
        parent: null,
    },
    {
        name: "../components/header.html",
        path: "/home/foo/example/src/html/components/header.html",
        parent: "/home/foo/example/src/html/screens/layout.html",
    },
    {
        name: "../components/footer.html",
        path: "/home/foo/example/src/html/components/footer.html",
        parent: "/home/foo/example/src/html/screens/layout.html",
    },
    {
        name: "./copyright.html",
        path: "/home/foo/example/src/html/components/copyright.html",
        parent: "/home/foo/example/src/html/components/footer.html",
    },
]
```

Pull request:
Emit 'load' events on Loader and Environment instances
## Summary

Adds a 'load' event to `Environment` and `Loader` instances, to allow runtime dependency detection.

Closes #1153

## Checklist

I've completed the checklist below to ensure I didn't forget anything. This makes reviewing this PR as easy as possible for the maintainers. And it gets this change released as soon as possible.

* [x] Proposed change helps towards [*purpose of this project*](https://github.com/mozilla/nunjucks/blob/master/CONTRIBUTING.md#purpose).
* [x] [*Documentation*](https://github.com/mozilla/nunjucks/tree/master/docs/) is added / updated to describe proposed change.
* [x] [*Tests*](https://github.com/mozilla/nunjucks/tree/master/tests) are added / updated to cover proposed change.
* [x] [*Changelog*](https://github.com/mozilla/nunjucks/blob/master/CHANGELOG.md) has an entry for proposed change (if user-facing fix or feature).


Patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 83907ab..5f44151 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,6 +1,9 @@
 Changelog
 =========
 
+* Emit 'load' events on `Environment` instances, to allow runtime dependency
+  tracking. Fixes [#1153](https://github.com/mozilla/nunjucks/issues/1153).
+
 3.1.7 (Jan 12 2019)
 ------------------
 
diff --git a/docs/api.md b/docs/api.md
index 2bf4f35..8c516a5 100644
--- a/docs/api.md
+++ b/docs/api.md
@@ -353,6 +353,23 @@ SafeString (to be documented) if one was passed in, so the output will
 copy the safeness of the input, but this property is helpful in rare
 circumstances.
 {% endapi %}
+
+{% api %}
+'load' event
+env.on('load', function(name, source, loader))
+
+The 'load' event gets emitted whenever a Loader retrieves the source of a
+template. It can be listened to in order to determine template dependencies
+at runtime. The arguments emitted to the callback are:
+
+* **name** *(String)* The template name, as passed to the loader
+* **source** *(Object)* The object that gets returned from Loader.getSource
+  * **src** *(String)* The template source
+  * **path** *(String)* The full path to the template
+  * **noCache** *(Bool)* If `true`, the template wasn't cached.
+* **loader** The Loader instance that triggered the event.
+{% endapi %}
+
 {% raw %}
 
 ## Template
diff --git a/docs/fr/api.md b/docs/fr/api.md
index 847ff34..babac4b 100644
--- a/docs/fr/api.md
+++ b/docs/fr/api.md
@@ -344,6 +344,22 @@ de rares circonstances.
 {% endapi %}
 {% raw %}
 
+{% api %}
+'load' événement
+env.on('load', function(name, source, loader))
+
+Quand une instance 'Loader' charge un template, elle émet l'événement
+'load'. On peut utiliser cette événement pour détérminer les dépendances
+à l'exécution. Ses arguments sont :
+
+* **name** *(String)* Le nom du template
+* **source** *(Object)* Le resultat de l'appel de `Loader.getSource`
+  * **src** *(String)* Le code source du template
+  * **path** *(String)* Le chemin d'accèss du fichier ou l'URL
+  * **noCache** *(Bool)* Si la valeur n'est pas mise en cache.
+* **loader** L'instance du Loader qui a émetté l'événement
+{% endapi %}
+
 ## Template
 
 Un `Template` est un objet qui gère la compilation des chaînes de template
diff --git a/nunjucks/src/compiler.js b/nunjucks/src/compiler.js
index 08b280f..431cde9 100644
--- a/nunjucks/src/compiler.js
+++ b/nunjucks/src/compiler.js
@@ -5,7 +5,7 @@ const transformer = require('./transformer');
 const nodes = require('./nodes');
 const {TemplateError} = require('./lib');
 const {Frame} = require('./runtime');
-const Obj = require('./object');
+const {Obj} = require('./object');
 
 // These are all the same for now, but shouldn't be passed straight
 // through
diff --git a/nunjucks/src/environment.js b/nunjucks/src/environment.js
index bb1c3b9..1cd7537 100644
--- a/nunjucks/src/environment.js
+++ b/nunjucks/src/environment.js
@@ -8,7 +8,7 @@ const filters = require('./filters');
 const {FileSystemLoader, WebLoader, PrecompiledLoader} = require('./loaders');
 const tests = require('./tests');
 const globals = require('./globals');
-const Obj = require('./object');
+const {Obj, EmitterObj} = require('./object');
 const globalRuntime = require('./runtime');
 const {handleError, Frame} = globalRuntime;
 const expressApp = require('./express-app');
@@ -37,7 +37,7 @@ const noopTmplSrc = {
   }
 };
 
-class Environment extends Obj {
+class Environment extends EmitterObj {
   init(loaders, opts) {
     // The dev flag determines the trace that'll be shown on errors.
     // If set to true, returns the full trace from the error point,
@@ -82,7 +82,7 @@ class Environment extends Obj {
       );
     }
 
-    this.initCache();
+    this._initLoaders();
 
     this.globals = globals();
     this.filters = {};
@@ -95,18 +95,28 @@ class Environment extends Obj {
     lib._entries(tests).forEach(([name, test]) => this.addTest(name, test));
   }
 
-  initCache() {
-    // Caching and cache busting
+  _initLoaders() {
     this.loaders.forEach((loader) => {
+      // Caching and cache busting
       loader.cache = {};
       if (typeof loader.on === 'function') {
-        loader.on('update', (template) => {
-          loader.cache[template] = null;
+        loader.on('update', (name, fullname) => {
+          loader.cache[name] = null;
+          this.emit('update', name, fullname, loader);
+        });
+        loader.on('load', (name, source) => {
+          this.emit('load', name, source, loader);
         });
       }
     });
   }
 
+  invalidateCache() {
+    this.loaders.forEach((loader) => {
+      loader.cache = {};
+    });
+  }
+
   addExtension(name, extension) {
     extension.__name = name;
     this.extensions[name] = extension;
diff --git a/nunjucks/src/loader.js b/nunjucks/src/loader.js
index 1ff3b20..139939e 100644
--- a/nunjucks/src/loader.js
+++ b/nunjucks/src/loader.js
@@ -1,23 +1,9 @@
 'use strict';
 
 const path = require('path');
-const Obj = require('./object');
-
-module.exports = class Loader extends Obj {
-  on(name, func) {
-    this.listeners = this.listeners || {};
-    this.listeners[name] = this.listeners[name] || [];
-    this.listeners[name].push(func);
-  }
-
-  emit(name, ...args) {
-    if (this.listeners && this.listeners[name]) {
-      this.listeners[name].forEach((listener) => {
-        listener(...args);
-      });
-    }
-  }
+const {EmitterObj} = require('./object');
 
+module.exports = class Loader extends EmitterObj {
   resolve(from, to) {
     return path.resolve(path.dirname(from), to);
   }
diff --git a/nunjucks/src/node-loaders.js b/nunjucks/src/node-loaders.js
index 4e3d6f9..42405a0 100644
--- a/nunjucks/src/node-loaders.js
+++ b/nunjucks/src/node-loaders.js
@@ -45,7 +45,7 @@ class FileSystemLoader extends Loader {
       watcher.on('all', (event, fullname) => {
         fullname = path.resolve(fullname);
         if (event === 'change' && fullname in this.pathsToNames) {
-          this.emit('update', this.pathsToNames[fullname]);
+          this.emit('update', this.pathsToNames[fullname], fullname);
         }
       });
       watcher.on('error', (error) => {
@@ -76,11 +76,13 @@ class FileSystemLoader extends Loader {
 
     this.pathsToNames[fullpath] = name;
 
-    return {
+    const source = {
       src: fs.readFileSync(fullpath, 'utf-8'),
       path: fullpath,
       noCache: this.noCache
     };
+    this.emit('load', name, source);
+    return source;
   }
 }
 
diff --git a/nunjucks/src/nodes.js b/nunjucks/src/nodes.js
index 65fd2fa..20c60d5 100644
--- a/nunjucks/src/nodes.js
+++ b/nunjucks/src/nodes.js
@@ -1,6 +1,6 @@
 'use strict';
 
-const Obj = require('./object');
+const {Obj} = require('./object');
 
 function traverseAndCheck(obj, type, results) {
   if (obj instanceof type) {
diff --git a/nunjucks/src/object.js b/nunjucks/src/object.js
index 39c49ac..e3a34c6 100644
--- a/nunjucks/src/object.js
+++ b/nunjucks/src/object.js
@@ -1,6 +1,7 @@
 'use strict';
 
 // A simple class system, more documentation to come
+const EventEmitter = require('events');
 const lib = require('./lib');
 
 function parentWrap(parent, prop) {
@@ -59,4 +60,26 @@ class Obj {
   }
 }
 
-module.exports = Obj;
+class EmitterObj extends EventEmitter {
+  constructor(...args) {
+    super();
+    // Unfortunately necessary for backwards compatibility
+    this.init(...args);
+  }
+
+  init() {}
+
+  get typename() {
+    return this.constructor.name;
+  }
+
+  static extend(name, props) {
+    if (typeof name === 'object') {
+      props = name;
+      name = 'anonymous';
+    }
+    return extendClass(this, name, props);
+  }
+}
+
+module.exports = { Obj, EmitterObj };
diff --git a/nunjucks/src/parser.js b/nunjucks/src/parser.js
index 45d0dc5..a98142c 100644
--- a/nunjucks/src/parser.js
+++ b/nunjucks/src/parser.js
@@ -2,7 +2,7 @@
 
 var lexer = require('./lexer');
 var nodes = require('./nodes');
-var Obj = require('./object');
+var Obj = require('./object').Obj;
 var lib = require('./lib');
 
 class Parser extends Obj {
diff --git a/nunjucks/src/web-loaders.js b/nunjucks/src/web-loaders.js
index c592aa5..609e588 100644
--- a/nunjucks/src/web-loaders.js
+++ b/nunjucks/src/web-loaders.js
@@ -45,6 +45,7 @@ class WebLoader extends Loader {
           path: name,
           noCache: !useCache
         };
+        this.emit('load', name, result);
         if (cb) {
           cb(null, result);
         }


Test patch:
diff --git a/tests/api.js b/tests/api.js
index 17d644a..275f2ed 100644
--- a/tests/api.js
+++ b/tests/api.js
@@ -82,5 +82,14 @@
         path: path.resolve(templatesPath, 'string.njk')
       })).to.be('FooTest3BazFizzle');
     });
+
+    it('should emit "load" event on Environment instance', function(done) {
+      var env = new Environment(new Loader(templatesPath));
+      env.on('load', function(name, source) {
+        expect(name).to.equal('item.njk');
+        done();
+      });
+      env.render('item.njk', {});
+    });
   });
 }());
diff --git a/tests/loader.js b/tests/loader.js
index 4c31428..c3b9a58 100644
--- a/tests/loader.js
+++ b/tests/loader.js
@@ -22,21 +22,6 @@
   }
 
   describe('loader', function() {
-    it('should have default opts for WebLoader', function() {
-      var webLoader = new WebLoader(templatesPath);
-      expect(webLoader).to.be.a(WebLoader);
-      expect(webLoader.useCache).to.be(false);
-      expect(webLoader.async).to.be(false);
-    });
-
-    if (typeof FileSystemLoader !== 'undefined') {
-      it('should have default opts for FileSystemLoader', function() {
-        var fileSystemLoader = new FileSystemLoader(templatesPath);
-        expect(fileSystemLoader).to.be.a(FileSystemLoader);
-        expect(fileSystemLoader.noCache).to.be(false);
-      });
-    }
-
     it('should allow a simple loader to be created', function() {
       // From Docs: http://mozilla.github.io/nunjucks/api.html#writing-a-loader
       // We should be able to create a loader that only exposes getSource
@@ -82,5 +67,49 @@
         done();
       });
     });
+
+    describe('WebLoader', function() {
+      it('should have default opts for WebLoader', function() {
+        var webLoader = new WebLoader(templatesPath);
+        expect(webLoader).to.be.a(WebLoader);
+        expect(webLoader.useCache).to.be(false);
+        expect(webLoader.async).to.be(false);
+      });
+
+      it('should emit a "load" event', function(done) {
+        var loader = new WebLoader(templatesPath);
+
+        if (typeof window === 'undefined') {
+          this.skip();
+        }
+
+        loader.on('load', function(name, source) {
+          expect(name).to.equal('simple-base.njk');
+          done();
+        });
+
+        loader.getSource('simple-base.njk');
+      });
+    });
+
+    if (typeof FileSystemLoader !== 'undefined') {
+      describe('FileSystemLoader', function() {
+        it('should have default opts', function() {
+          var loader = new FileSystemLoader(templatesPath);
+          expect(loader).to.be.a(FileSystemLoader);
+          expect(loader.noCache).to.be(false);
+        });
+
+        it('should emit a "load" event', function(done) {
+          var loader = new FileSystemLoader(templatesPath);
+          loader.on('load', function(name, source) {
+            expect(name).to.equal('simple-base.njk');
+            done();
+          });
+
+          loader.getSource('simple-base.njk');
+        });
+      });
+    }
   });
 }());