Repository: highlightjs/highlight.js (JavaScript)

Issue:
highlightjs.org build seems to differ from `./tools/build.js -t browser :common`
**Expected behavior: The build would be identical to building locally using the build process.**

Copying from https://github.com/highlightjs/highlight.js/issues/1245

> I've asked few times before, where is the code which is responsible for creating the download and/or concatenating the fragments into a single download?

You'd think it was the build process, but it seems the results are different. Only @isagalaev knows I think. The code generated on the website is putting hljs.registerLanguage OUTSIDE the original enclosure and the code generated by the build process places it inside. @isagalaev Is the website using a special build system instead of the one in the repo for some reason? hljs as a global only works in the web environment (where anything on window is treated as a global), so that is why it's failing in node.js.
Require using AMD broken in 10.0.0
Loading via SystemJS in AMD used to work with v9.18.1. Broken in v10:

```
system.js:4 Uncaught (in promise) ReferenceError: hljs is not defined
    at eval (highlight.js:1566)
    at eval (highlight.js:6322)
    at eval (<anonymous>)
    at Qe (system.js:4)
    at system.js:4
    at async Promise.all (:63342/Kidoju.Widgets/src/js/widgets/index 10)
```

hljs is undefined when calling  L1566

```
hljs.registerLanguage('apache', function () {
   ...
}());
```

In System.config.js, I have:

```
packages: {
    'src/js': {
        defaultExtension: 'js',
        meta: {
           '*.js': {
                babelOptions: {
                    es2015: false,
                    stage2: false,
                    stage3: false,
                },
                format: 'amd',
            },
            '*.es6': {
                format: 'esm',
            },
            '*.jsx': {
                format: 'esm',
            },
            '*.mjs': {
                format: 'esm',
            },
         },
      },
 },
```

A workaround for now is to add an exception in meta:

```
 'vendor/highlight/highlight.js': {
      // Note: AMD broken as of v10
      format: 'global',
},
```


Discussion:
This will become even more important as we move forward with changes to the codebase and build system... the website needs to be using the same exactly build system we are using for development work and continuous integration or else this problem is going to get worse in the near future - with no guarantee that the builds the website produces are even usable at all.
@isagalaev Ping.
@yyyc514 I've been missing out on my bug mail lately… Life stuff, as usual. And I do remember I wanted to publish/open up the site's code, but haven't gotten to it yet.

Now, onto the build stuff. The site doesn't use `build.js -t browser` because it's supposed to build a custom package according to check boxes, and building it from scratch every time is too expensive. So instead I do this:

- upon a new release the site runs `build.js -t cdn` and stores all pre-build language files in a directory, and then `build.js -t cdn none` to create a pre-build core without any language files and stores it too
- upon a download request it concatenates them into a single piece and zips it on the fly, which is fast

So there is a (undocumented) assumption that a browser build is basically just a concatenation of files stored on a CDN.

Did `build.js -t browser` change recently which could break this assumption?
> So there is a (undocumented) assumption that a browser build is basically just a concatenation of files stored on a CDN.

Obviously true, but I'd say long term that is a faulty assumption.  It prevents us from doing useful things (for a monolith build) like dead-code/constant elimination, allowing languages to share code (at compile time via ES6 modules), etc...

> Did build.js -t browser change recently which could break this assumption?

Not significantly, not yet. 

But, I've been playing with ES6 module based builds that import everything in one file and then compile/minify/terser that result, and it's also (not surprising) smaller than just concat-ing a bunch of individual files with all the repeated `registerLanguage` boilerplate.

So I'd like to see the build process continue to improve... I suppose we could just accept that the source build scripts were "better" than the build the website gives you, but then we're introducing a fourth deliverable (CDN, browser, node, and now WEBSITE browser build)- and there does seem to be some expectation from users (not wholly unreasonable) that building from `git` and building on the website would produce identical output.



I agree with the general gist. I'd like to eventually get rid of the site code imitating the build process. However I have a pretty strong feeling that running an actual build process for every download would not be feasible. Which means that we need to get rid of the whole idea of custom builds made by the site. Unless I'm missing another option?

In fact I've been waiting for technology, namely HTTP/2, to mature enough to not even need a one-file distribution, custom or not. So that most browser users would do something like:

```html
<script src="{cdn-path/highlight.js}"></script>
<script>
hljs.initHighlightingOnLoad('html', 'css', 'javascript', ...);
</script>
```

The last line might generate separate `<script>` tags referring to individual languages or do some fancy module loading (I have no idea if modern browsers can finally do it).

If we could do that then we can deprecate the "browser" build altogether and just tell people to use whatever packing solution they like.
> However I have a pretty strong feeling that running an actual build process for every download would not be feasible. 

Well, without seeing the requests per minute numbers of peak traffic it's hard to say, but I for sure understand your concerns with the old build process being so slow.  I'm not sure the newer one is much faster.

> Which means that we need to get rid of the whole idea of custom builds made by the site. Unless I'm missing another option?

I do think if the server-side were all Node.js (not Python) that you might have some options with caching the intermediate state that Terser/Rollup allows you to do, but really I'm just spitballing.  I know that can help a lot with Rollup but I'm not sure it can help a lot with Terser (or if it's an option).

> If we could do that then we can deprecate the "browser" build altogether and just tell people to use whatever packing solution they like.

Time will tell I suppose. :-)
I think for the present I see two obvious paths:

- Cripple our browser build to pretty much duplicate the current "concat" strategy.  True, it's not currently a lot different presently, but I'd had high hopes we could do some creative things with it moving forward. 
- Potentially add a note that the website build is a "convenience" build and that you get the tightest/most optimized build by building directly from source.

> you might have some options with caching

You know, I looked at the stats now, and I think it can actually work. Out of 327 downloads, 277 (85%) is the default set, with the rest of them being mostly unique combinations of languages. I think users won't be too upset if downloading a unique combination takes several seconds. I'll look into it.

> Potentially add a note that the website build is a "convenience" build and that you get the tightest/most optimized build by building directly from source.

Yes, that's a good interim solution. As long as the concatenated result actually works :-)
> Yes, that's a good interim solution. As long as the concatenated result actually works :-)

Well, it *should* so long as we don't break the basic CDN/registration stuff.  The one reported issue so far is that the website build can't be used by node, only in the browser, but we've decided that's a non-issue as the intended target of the website build is the web, not node.  
> You know, I looked at the stats now, and I think it can actually work. Out of 327 downloads, 277 (85%) is the default set

Not even what I meant. :-) But yes caching the "whole shebang" can help also.  I meant that most of these build libs have internal caches that can make doing the work "repeatedly" much faster... but that's usually assuming you have a long running process.  So if you're running Python and only spawning short-running node.js processes for building that might not be any help - even if it would speed things up.


> So if you're running Python and only spawning short-running node.js processes for building that might not be any help

Ah, got you now.
@isagalaev Have you tested our website build for Web Worker usage?  I don't think it will work because the worker doesn't have `window` and therefore `hljs` isn't available to register languages... In my own modified build I add the following *BETWEEN* the core-lib and the language modules:

```js
if (typeof importScripts === 'function') { var hljs = self.hljs; }
```

IE:

```
var workerStub = "if (typeof importScripts === 'function') { var hljs = self.hljs; }";

  var fullSrc = [
    header, libraryCore, workerStub,
    ...languages.map((lang) => lang.moduleSrc) ].join("\n");
```

Could we make that fix?  Is that a reasonable resolution you think?  It's certainly a quick fix.  
@isagalaev Actually the new UMD wrapper code uses `this`, not `self` which seems to "just work" with workers, so I've removed the stub as it's no longer required and that solves that problem...

So can we decide a path forward here? If it's not possible for the website to run the actual build process (which is only going to get better/more tightly integrated with time) can we add a comment saying that for the "most optimized" build one should build directly from source?

We can work on the verbiage... I'm just trying to figure out the "next steps" for some of the more nebulous issues we have open still.

CC @egor-rogov 
> can we add a comment saying that for the "most optimized" build one should build directly from source?

The new README already does this, but is its still worth adding a mention to the website also?
@isagalaev Given any more thought to this?  I could close this with https://github.com/highlightjs/highlight.js/pull/2511 if we really wanted.  Just switching to a global period and then exporting that with `module.exports` means the website builds can now be used for the web OR required directly with Node.

There are still build improvements in the future that wouldn't allow the website to have access to, but for now it would mean parity.

**I'm not sure we ever were truly AMD compatible**, the old init just did this:

```js
  globalObject.hljs = factory({});

    // Finally register the global hljs with AMD.
    if(typeof define === 'function' && define.amd) {
      define([], function() {
        return globalObject.hljs;
      });
    }
```

IE, define a global, and then tell AMD to use that global... the new build system actually is actually behaving better (AMD wise) but the problem is now when we're trying to register the language modules there is no top-level `hljs` yet likely because I would assume AMD isn't (ever?) going to make one for us?

**The language modules have always registered themselves via a top-level `hljs` global object.**

So the behavior is different, but I'm not sure that's actually a problem - since it was "all globals all along" so far as I can tell. I'm not sure what the right answer is here because I don't think what we're doing before is actually correct... so perhaps someone with more knowledge can weigh in here.


Perhaps we should just remove our AMD support?  Perhaps someone super familiar with AMD could explain what we'd have to do with the language modules to make them truly AMD compatible (hard to say if that's viable without knowing what would be required)... just going back to how we had it before seems quite broken to me.

Do all the language modules need UMD wrappers? That would be annoying.

Some language modules also have their own built-in dependencies and expect to be loaded in the proper sequence... they would not do so well if the order were randomized, etc. (the build process takes care of this by inserting the registrations in the proper order in the final web build product).
I think the problem is in the structure of the build output file which is:

```
(function (global, factory) {
  typeof exports === 'object' && typeof module !== 'undefined' ? module.exports = factory() :
  typeof define === 'function' && define.amd ? define(factory) :
  (global = global || self, global.hljs = factory());
}(this, (function () { 'use strict';
  const HLJS = function(hljs) {
    ....
  }
  var highlight = HLJS({});
  return highlight;
})));

hljs.registerLanguage('apache', function () { // <-- hljs is undefined when using AMD
  ...
}());
```

After removing all `hljs.registerLanguage`, AMD loading works fine and returns the hljs object to the loader.
Yes, I understand exactly why it is failing.  **The new AMD code fails to setup a global object (as it did before), so this now fails.**

I'm less sure of the path forward.

This output is how the build process works (and always has).  Languages are "modules" that are added AFTER the core library is setup by calling `registerLanguage`.  This is also  currently a requirement of the website builder/download tool too... the languages are just appeneded to the end of the core library.


> After removing all hljs.registerLanguage, AMD loading works fine and returns the hljs object to the loader.

And now you have no languages, so the highlighter won't work. :)
From a different angle: What benefits do you gain by using AMD here vs having to declare it "global"?
I know but the problem is not with your UMD wrapper which is fine.
> I know but the problem is not with your UMD wrapper which is fine.

Well it depends how you define the problem. :-)  You could also say the wrapper is EXACTLY the problem.  The old wrapper setup a global, the new wrapper does not - and as a result it fails.  

@egor-rogov Any thoughts here?
From my end, I can certainly use a workaround either with SystemJS (development) or Webpack (production). I'll always find a solution. I just thought it was nice to report an issue.
> I just thought it was nice to report an issue.

It certainly was. :-)  And thanks.  I'm just wondering the value of saying we support AMD when truly we don't and we register a global anyways... I have a feeling doing it "properly" might be time and effort better spent on other things...

Isn't part of the point of AMD to control/defer executing the JS modules?

Since Highlight.js has no external requirements I'm starting to wonder what AMD gives anyone here anyways?  I guess you could express that other parts of your system are dependent on Highlight.js?
The issue is you are mixing modules and code outside of modules in the same file. I have also tried loading via CommonJS: same error:

```
Uncaught (in promise) ReferenceError: hljs is not defined
    at Object.eval (highlight.js:1566)
    at eval (highlight.js:6322)
    at eval (highlight.js:6323)
    at eval (<anonymous>)
    at Qe (system.js:4)
    at system.js:4
    at T (system.js:4)
    at U (system.js:4)
    at system.js:4
    at system.js:4
```

For this, simply configure SystemJS with:

```
'vendor/highlight/highlight.js': {
    format: 'cjs',
},
```

Your npm module works because it does not use the same code as the build output, highlight.pack.js. The npm module actually loads the module before loading languages:

```
var hljs = require('./core');
hljs.registerLanguage('apache', require('./languages/apache'));
```

In highlight.pack.js, the module id defined but not loaded before loading language packs, unless it has been assigned to global.

The cleaner option which I have just tested and which works is to change the structure as follows:

```
(function (global, factory) {
  typeof exports === 'object' && typeof module !== 'undefined' ? module.exports = factory() :
  typeof define === 'function' && define.amd ? define(factory) :
  (global = global || self, global.hljs = factory());
}(this, (function () { 'use strict';
  const HLJS = function(hljs) {
    ....
  }
  var highlight = HLJS({});

  highlight.registerLanguage('apache', function () {
      ...
  }());

  return highlight;
})));
```

This concatenates the file a bit differently but the result works both with AMD and CJS in addition to global = true UMD!

I have attached the working file for a diff: [highlight.zip](https://github.com/highlightjs/highlight.js/files/4530789/highlight.zip)
But that’s currently not possible for the website builder and would also break 3rd party languages that load later and expect a global. So if we are going to solve this we can’t just solve it half way. 

I think we hack it like we did before or else it seems all the language modules need to become UMD. I’m not even sure how that might work. Surely AMD supports named dependencies?
Do you know what the proper AMD way would be to call the registration at a later point?  

Ie, how do I get the internal HLJS object back out of Amd or pass a callback that can access it?
So if we give our module an id (which is easy enough):

```js
typeof define === 'function' && define.amd ? define('highlight.js', factory) :
```

Then I think later you could do:

```js

if (typeof define === 'function' && define.amd) {
  define(null, ["highlight.js"], (hljs) => { 
   hljs.registerLanguage('apache',...
  });
} else {
   hljs.registerLanguage('apache'...
}
```
Which is kind of what I expected... But I'm wondering what we gain by jumping thru all those hoops since no one has ever seemed to mind a global in the past?

Doing this would require all the 3rd party distributables to be recompiled to also include a UMD wrapper...
@yyyc514,

IMHO:
1.  UMD loading in highlight.pack.js v10 is broken both for AMD and CommonJS,
2. highlight is a nice piece of code and you cannot have a broken feature: fix it or remove it,
3. your suggestion here above, is not the proper way to fix it for AMD and it does not fix it for CommonJS (I could explain but I am not sure it is worth the trouble).
4. In the long term, modules are now part of ECMAScript and are the way forward,
5. in the short term, most loaders including webpack can accommodate globals,
6. so in the end this is a question of product roadmap driven by user requirements.

I would recommend:
1. Until you have a proper fix, stick to global only to avoid misleading your users with features that do not work;
2. Give me a few days to look at the code for building highlight.pack.js and see if I can come up with an easy suggestion to fix this. If I can, I'll make a PR.


I just realized there is a lot of custom code to build highligh.js but why reinvent the wheel? i.e. why not use webpack to build your library? cf. https://webpack.js.org/guides/author-libraries/#expose-the-library. Any specific requirements? Just asking as you might consider delaying UMD as part of your product roadmap.
> @egor-rogov Any thoughts here?

No idea. Until today I was pretty sure that AMD is for Advanced Micro Devices (:
> Youur suggestion here above, is not the proper way to fix it for AMD 

I was talking solely about how to do it "best" for AMD above... is that not how you would do it? Register the library with an ID and then make that id a dependency of the languages (which would also load via AMD)?  Remember we have to support languages loaded at any time, not just those bundled with the JS file.

> highlight.pack.js

Do you mean `.min.js`? There is no pack file anymore with version 10.

> In the long term, modules are now part of ECMAScript and are the way forward,

If someone wanted to use the **raw** source it's all modules now and could be hooked into whatever build system, but the web library is built to a single simple file and the NPM library is built to CommonJS (require, etc), which Node still seems to be stuck on...

> and CommonJS,

_I'm not sure the web build has ever been CommonJS_ compatible (haha, and looks like it was because of how it bundled the languages - though i'm not sure the versions build by the website ever were), but feel free to correct me if I'm mistaken.  Previous `src/highlight.js` happened to be since it ONLY included the library code, but that doesn't make sense now since that file is ES6 modules now, not a built file...

If someone wants to use CommonsJS they need to use the NPM package we provide.
> In the long term, modules are now part of ECMAScript and are the way forward,

Agree, and when Node.js natively supports them broadly then I think we'd look at publishing an NPM package that was ES6 modules vs require... until then anyone is free to use the raw source which is ahead of the curve (since v10).
> I just realized there is a lot of custom code to build highligh.js but why reinvent the wheel? i.e. why not use webpack to build your library? cf. https://webpack.js.org/guides/author-libraries/#expose-the-library. Any specific requirements? Just asking as you might consider delaying UMD as part of your product roadmap.

We have very specific requirements and the JS build code is all quite clear and easy to follow IMHO - it's all just functions, very little magic.  You only need to know Javascript (and a bit of rollup).  If we simplify it further in the future it'll likely be on top of rollup, not webpack. I've never cared for webpack; never seemed simple or intuitive to me at all.

The goal was never to ship a UMD build, just rather to "preserve" the existing AMD functionality from before... sadly I didn't realize how "hacked" it was, so we failed to do that. 

---

If you want to explain how you feel the "proper" way to do AMD in this situation is, I will consider it but at this point I think the best way forward may be to just drop the previous hacky AMD support and not pretend to do AMD at all - and *for the browser* just ship a global version - which is really what we'd done all along anyways.
So if we just fallback to IIFE for the browser build we'll have:

```js
/*
  Highlight.js 10.0.0 (3875088a)
  License: BSD-3-Clause
  Copyright (c) 2006-2020, Ivan Sagalaev
*/
var hljs = (function () {
  'use strict';
  ...
```

Just a top level variable...
@jlchereau Out of curiosity how were you using HLJS before?  Just the default build with "common" languages built-in via AMD?

What will people's AMD pipelines do if we just switch to a pure global? Will they need to change their configs or will it just work despite not really being AMD?
@jlchereau Can you take a quick look at the PR.  It falls back to an IIFE build but also hooks up the variable to module.exports if it's around.  I think this:

- Works in the browser (IIFE, global var `hljs`)
- Works for CommonJS (my Node.js test file can requite it and use it just fine)
- (therefore restoring the ability for someone to just grab the primary distributable and use it standalone with Node.js if they really wanted to)
- Drops support for AMD which we never truly supported properly anyways
@jlchereau Any further thoughts?  If not I'm going to move forward with just supporting CommonJS and IIFE/global and drop support for AMD.



Patch:
diff --git a/.travis.yml b/.travis.yml
index fa61a6a0e9..67fd1364cf 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -29,5 +29,7 @@ script:
       npm run test
     else
       npm run test-browser
+      # our browser build should also work fine as a Node.js CommonJS module
+      node test/builds/browser_build_as_commonjs.js
     fi
 sudo: false  # Use container-based architecture
diff --git a/CHANGES.md b/CHANGES.md
index 711f466e41..8d9201c218 100644
--- a/CHANGES.md
+++ b/CHANGES.md
@@ -30,6 +30,22 @@ Language Improvements:
 [Vania Kucher]: https://github.com/qWici
 
 
+## Version 10.0.2 (pending)
+
+Brower build:
+
+- [Issue](https://github.com/highlightjs/highlight.js/issues/2505) (bug) Fix: Version 10 fails to load as CommonJS module. (#2511) [Josh Goebel][]
+- [Issue](https://github.com/highlightjs/highlight.js/issues/2505) (removal) AMD module loading support has been removed. (#2511) [Josh Goebel][]
+
+
+Parser Engine Changes:
+
+- ...
+
+
+[Josh Goebel]: https://github.com/yyyc514
+
+
 ## Version 10.0.1
 
 Parser Engine Changes:
diff --git a/tools/build_config.js b/tools/build_config.js
index 6ce035be96..c32efe993c 100644
--- a/tools/build_config.js
+++ b/tools/build_config.js
@@ -29,7 +29,8 @@ module.exports = {
         },
         output: {
           name: "hljs",
-          format: "umd",
+          format: "iife",
+          footer: "if (typeof exports === 'object' && typeof module !== 'undefined') { module.exports = hljs; }",
           interop: false,
         }
       },


Test patch:
diff --git a/test/builds/browser_build_as_commonjs.js b/test/builds/browser_build_as_commonjs.js
new file mode 100644
index 0000000000..4315df2ff0
--- /dev/null
+++ b/test/builds/browser_build_as_commonjs.js
@@ -0,0 +1,6 @@
+const hljs = require("../../build/highlight");
+
+let major = parseInt(majorVersion=hljs.versionString.split("."))
+if (major != 10) {
+  process.exit(1)
+}