Repository: ProjectEvergreen/greenwood (js)

Issue:
npx commands showing error for not found node modules paths
## Type of Change
- [ ] New Feature Request
- [ ] Documentation / Website
- [ ] Improvement / Suggestion
- [x] Bug
- [ ] Other (please clarify below)

## Summary
When running the `npx` commands for using Greenwood, there are some errors for files not found in _node_modules_,

ex.
```sh
% npx @greenwood/cli serve
Downloading Chromium r818858 - 132.4 Mb [====================] 100% 0.0s
-------------------------------------------------------
Welcome to Greenwood ♻️
-------------------------------------------------------
Running Greenwood with the serve command.
Initializing project config
Initializing project workspace contexts
Generating graph of workspace files...
Started local development server at localhost:1984
Serializing pages at http://127.0.0.1:1984
pages to generate
 src/pages/index.md
serializing page... /

  Error: ENOENT: no such file or directory, open '/Users/owenbuckley/Workspace/scratch/node_modules/es-module-shims/dist/es-module-shims.js'


  Error: ENOENT: no such file or directory, open '/Users/owenbuckley/Workspace/scratch/node_modules/@webcomponents/webcomponentsjs/webcomponents-bundle.js'

Serializing complete for page /.
done serializing all pages
@rollup/plugin-replace: 'preventAssignment' currently defaults to false. It is recommended to set this option to `true`, as the next major version will default this option to `true`.
copying graph.json...
Initializing project config
Initializing project workspace contexts
Generating graph of workspace files...
Started production test server at localhost:8080

  Error: ENOENT: no such file or directory, open '/Users/owenbuckley/Workspace/scratch/node_modules/es-module-shims/dist/es-module-shims.js'


  Error: ENOENT: no such file or directory, open '/Users/owenbuckley/Workspace/scratch/node_modules/@webcomponents/webcomponentsjs/webcomponents-bundle.js'
```

## Details
This is likely due to the fact that certain dependencies are assumed to be in the user's current directory where a _node_modules_ directory would be.  When running with `npx` then this wouldn't exist, so how to resolve these???

![Screen Shot 2021-04-03 at 6 46 06 PM](https://user-images.githubusercontent.com/895923/113524200-3ead6880-957b-11eb-9ea8-7bd6a004d7b4.png)

Pull request:
Bug/issue 505 improve default npx support
<!--
## Submitting a Pull Request
We love contributions and appreciate any help you can offer!
-->

## Related Issue
resolves #505 

## Summary of Changes
1. Add fallback detection to unpkg.com if internal client side dependencies for **es-module-shims** and **@webcomponents/webcomponentjs** aren't found in `process.cwd()`
1. Add `npx` docs back to the home page and quick start
1. Add a test specifically to omit those dependencies from scaffolding (is just a copy / paste of default case)
1. Whitelist **unpkg.com** with puppeteer

## TODO
1. [x] noticed a bug when trying to merge user page templates while still using default app template, so will want to make a bug and fix that first, then can enable the `xdescribe` spec here and then we can merge.
1. [x] rebase after #513 is merged
 
## Notes
For now, just getting this quick and dirty happy path working, using unpkg.com as a fallback.  But I did reach out in NodeJS slack channel and got this tip on using NodeJS [`require.resolve`](https://nodejs.org/api/modules.html#modules_require_resolve_request_options).

Essentially, give it a package name and it will look up the path for you.
```js
// /Users/owenbuckley/Workspace/project-evergreen/repos/greenwood/packages/cli/node_modules/es-module-shims/dist/es-module-shims.js
console.debug( require.resolve('es-module-shims') )
```

With a better local testing environment (using `npm|yarn link`), we could set it up so that it in **plugins-node-modules** `resolve` just delegates all this to `require.resolve` instead:
```js
// example url => /node_modules/es-module-shims/dist/es-module-shims.js
async resolve(url) {
  return new Promise((resolve, reject) => {
    try {
      const packageName = url.split('/')[1]; // ex. es-module-shims
      const packageFullPath = require.resolve(packageName); // ex. /Users/.../node_modules/es-module-shims/dist/es-module-shims.js
      const currentDirectory = packageFullPath.split('/node_modules/')[0]; //ex. /Users/.../
      const nodeModulesUrl = path.join(currentDirectory, url.replace.replace('/node_modules/'); //ex. /Users/.../es-module-shims/dist/es-module-shims.js

      resolve(nodeModulesUrl);
    } catch (e) {
      console.error(e);
      reject(e);
    }
  });
}
```

We want to generate our own full path since the `url` is what we want based on walking _package.json_ (in theory) as opposed to what NodeJS _thinks_ we want.

----

That said, perhaps there is a case for us to entirely refactor [`getPackageEntryPath`](https://github.com/ProjectEvergreen/greenwood/blob/master/packages/cli/src/plugins/resource/plugin-node-modules.js#L14) to be based off `require.resolve` logic instead? 🤔 

We intentionally always want to favor ESM if we can, so I guess it depends on if anything would break but we certainly have enough tests for that.  😃 

I'll definitely be making a follow up issue for us to explore this more so we can try and provide the best `npx` experience possible as part of hitting 1.0, and not have to worry ourselves about all this _node_modules_ look up stuff if we can help it.

Patch:
diff --git a/packages/cli/src/lib/browser.js b/packages/cli/src/lib/browser.js
index 0ed78ff5..1d4e57e3 100644
--- a/packages/cli/src/lib/browser.js
+++ b/packages/cli/src/lib/browser.js
@@ -42,7 +42,8 @@ class BrowserRunner {
 
       if (
         interceptedRequestUrl.indexOf('http://127.0.0.1') >= 0 ||
-        interceptedRequestUrl.indexOf('localhost') >= 0
+        interceptedRequestUrl.indexOf('localhost') >= 0 ||
+        interceptedRequestUrl.indexOf('unpkg.com') >= 0
       ) {
         interceptedRequest.continue();
       } else {
diff --git a/packages/cli/src/plugins/resource/plugin-node-modules.js b/packages/cli/src/plugins/resource/plugin-node-modules.js
index bdeef471..b030b1a6 100644
--- a/packages/cli/src/plugins/resource/plugin-node-modules.js
+++ b/packages/cli/src/plugins/resource/plugin-node-modules.js
@@ -184,11 +184,6 @@ class NodeModulesResource extends ResourceInterface {
         const fullUrl = path.extname(url) === ''
           ? `${url}.js`
           : url;
-        // const fullUrl = path.extname(url) === ''
-        //   ? fs.existsSync(`${url}.mjs`) // test for .mjs first
-        //     ? `${url}.mjs`
-        //     : `${url}.js`
-        //   : url;
         const body = await fs.promises.readFile(fullUrl);
 
         resolve({
@@ -208,7 +203,7 @@ class NodeModulesResource extends ResourceInterface {
   async intercept(url, body) {
     return new Promise((resolve, reject) => {
       try {
-        const { userWorkspace } = this.compilation.context;
+        const { userWorkspace, projectDirectory } = this.compilation.context;
         let newContents = body;
         const hasHead = body.match(/\<head>(.*)<\/head>/s);
 
@@ -228,7 +223,11 @@ class NodeModulesResource extends ResourceInterface {
           : fs.existsSync(`${process.cwd()}/package.json`)
             ? require(path.join(process.cwd(), 'package.json'))
             : {};
-
+        const esShimsFilename = '/node_modules/es-module-shims/dist/es-module-shims.js';
+        const esShimsPath = fs.existsSync(path.join(projectDirectory, esShimsFilename))
+          ? esShimsFilename
+          : 'https://unpkg.com/es-module-shims@0.5.2/dist/es-module-shims.js';
+        
         // walk the project's pacakge.json for all its direct dependencies
         // for each entry found in dependencies, find its entry point
         // then walk its entry point (e.g. index.js) for imports / exports to add to the importMap
@@ -237,7 +236,7 @@ class NodeModulesResource extends ResourceInterface {
 
         newContents = newContents.replace('<head>', `
           <head>
-            <script defer src="/node_modules/es-module-shims/dist/es-module-shims.js"></script>
+            <script defer src="${esShimsPath}"></script>
             <script type="importmap-shim">
               {
                 "imports": ${JSON.stringify(importMap, null, 1)}
diff --git a/packages/cli/src/plugins/resource/plugin-standard-html.js b/packages/cli/src/plugins/resource/plugin-standard-html.js
index 982560d3..827051a9 100644
--- a/packages/cli/src/plugins/resource/plugin-standard-html.js
+++ b/packages/cli/src/plugins/resource/plugin-standard-html.js
@@ -143,11 +143,16 @@ const getAppTemplate = (contents, userWorkspace) => {
   return appTemplateContents;
 };
 
-const getUserScripts = (contents) => {
+const getUserScripts = (contents, projectDirectory) => {
   if (process.env.__GWD_COMMAND__ === 'build') { // eslint-disable-line no-underscore-dangle
+    const wcBundleFilename = '/node_modules/@webcomponents/webcomponentsjs/webcomponents-bundle.js';
+    const wcBundlePath = fs.existsSync(path.join(projectDirectory, wcBundleFilename))
+      ? wcBundleFilename
+      : 'https://unpkg.com/@webcomponents/webcomponentsjs@2.4.4/webcomponents-bundle.js';
+
     contents = contents.replace('<head>', `
       <head>
-        <script src="/node_modules/@webcomponents/webcomponentsjs/webcomponents-bundle.js"></script>
+        <script src="${wcBundlePath}"></script>
     `);
   }
   return contents;
@@ -210,7 +215,7 @@ class StandardHtmlResource extends ResourceInterface {
     return new Promise(async (resolve, reject) => {
       try {
         const config = Object.assign({}, this.compilation.config);
-        const { userWorkspace } = this.compilation.context;
+        const { userWorkspace, projectDirectory } = this.compilation.context;
         const normalizedUrl = this.getRelativeUserworkspaceUrl(url);
         let body = '';
         let template = null;
@@ -270,7 +275,7 @@ class StandardHtmlResource extends ResourceInterface {
         
         body = getPageTemplate(barePath, userWorkspace, template);
         body = getAppTemplate(body, userWorkspace);
-        body = getUserScripts(body);
+        body = getUserScripts(body, projectDirectory);
         body = getMetaContent(normalizedUrl, config, body);
         
         if (processedMarkdown) {
@@ -306,9 +311,9 @@ class StandardHtmlResource extends ResourceInterface {
         if (hasHead && hasHead.length > 0) {
           let contents = hasHead[0];
 
-          contents = contents.replace(/<script src="\/node_modules\/@webcomponents\/webcomponentsjs\/webcomponents-bundle.js"><\/script>/, '');
+          contents = contents.replace(/<script src="(.*webcomponents-bundle.js)"><\/script>/, '');
           contents = contents.replace(/<script type="importmap-shim">.*?<\/script>/s, '');
-          contents = contents.replace(/<script defer="" src="\/node_modules\/es-module-shims\/dist\/es-module-shims.js"><\/script>/, '');
+          contents = contents.replace(/<script defer="" src="(.*es-module-shims.js)"><\/script>/, '');
           contents = contents.replace(/type="module-shim"/g, 'type="module"');
 
           body = body.replace(/\<head>(.*)<\/head>/s, `
diff --git a/www/pages/getting-started/quick-start.md b/www/pages/getting-started/quick-start.md
index 58487232..81378824 100644
--- a/www/pages/getting-started/quick-start.md
+++ b/www/pages/getting-started/quick-start.md
@@ -22,7 +22,7 @@ $ npm start
 Done!
 
 
-<!-- Additionally, you can start your own project right now as easy as 1.. 2.. 3, right from the command line!
+Additionally, you can start your own project right now as easy as 1.. 2.. 3, right from the command line!
 ```bash
 # with NodeJS already installed
 # create a pages directory for your content
@@ -33,6 +33,6 @@ $ echo "## hello world" > src/pages/index.md
 
 # run one of Greenwood's commands, and that's it!
 $ npx @greenwood/cli develop
-``` -->
+```
 
 _To learn more about what you can do with Greenwood, head over to our [documentation](/docs/) or feel to review the other sections in this guide.  To setup a project from scratch, you can pick up the rest of the getting started guide in the [project setup](/getting-started/project-setup) page._
\ No newline at end of file
diff --git a/www/pages/index.html b/www/pages/index.html
index ae5de3bc..7e27f362 100644
--- a/www/pages/index.html
+++ b/www/pages/index.html
@@ -33,7 +33,7 @@
             </div>
 
             <div class="quickstart">
-              <!-- <h2>Quick Start Example</h2>
+              <h2>Quick Start Example</h2>
               <pre class="language-bash">
                 <code class="language-bash">
                   # with NodeJS already installed
@@ -46,7 +46,7 @@
                   # run one of Greenwood's commands, and that's it!
                   $ npx @greenwood/cli develop
                 </code>
-              </pre> -->
+              </pre>
               <p>For anyone new to Greenwood, we encourage you go through our <a href="/getting-started/">Getting Started</a> guide, or to get right to work, head on over to our <a href="/docs/">documentation</a>.</p>
             </div>
 


Test patch:
diff --git a/packages/cli/test/cases/build.default.quick-start-npx/build.default.quick-start-npx.spec.js b/packages/cli/test/cases/build.default.quick-start-npx/build.default.quick-start-npx.spec.js
new file mode 100644
index 00000000..f8a369d1
--- /dev/null
+++ b/packages/cli/test/cases/build.default.quick-start-npx/build.default.quick-start-npx.spec.js
@@ -0,0 +1,118 @@
+/*
+ * Use Case
+ * Run Greenwood build command with no config and emulating being run with npx
+ *
+ * User Result
+ * Should generate a bare bones Greenwood build with no errors for missing files 
+ * by specifically not scaffolding node_modules/ needed for es-modules-shims and webcomponents-bundle.
+ * https://github.com/ProjectEvergreen/greenwood/issues/505
+ *
+ * User Command
+ * greenwood build
+ *
+ * User Config
+ * None (Greenwood Default)
+ *
+ * User Workspace
+ * Greenwood default (src/)
+ */
+const expect = require('chai').expect;
+const { JSDOM } = require('jsdom');
+const path = require('path');
+const runSmokeTest = require('../../../../../test/smoke-test');
+const TestBed = require('../../../../../test/test-bed');
+
+describe('Build Greenwood With: ', function() {
+  const LABEL = 'Default Greenwood Configuration and Workspace and emulating npx';
+  let setup;
+
+  before(async function() {
+    setup = new TestBed();
+    this.context = await setup.setupTestBed(__dirname, [], false);
+  });
+
+  describe(LABEL, function() {
+
+    before(async function() {
+      await setup.runGreenwoodCommand('build');
+    });
+    
+    runSmokeTest(['public', 'index'], LABEL);
+  
+    describe('Default output for index.html', function() {
+      let dom;
+
+      before(async function() {
+        dom = await JSDOM.fromFile(path.resolve(this.context.publicDir, './index.html'));
+      });
+
+      xdescribe('head section tags', function() {
+        let metaTags;
+
+        before(function() {
+          metaTags = dom.window.document.querySelectorAll('head > meta');
+        });
+
+        it('should have a <title> tag in the <head>', function() {
+          const title = dom.window.document.querySelector('head title').textContent;
+    
+          expect(title).to.be.equal('My App');
+        });
+
+        it('should have five default <meta> tags in the <head>', function() {
+          expect(metaTags.length).to.be.equal(5);
+        });
+
+        it('should have default charset <meta> tag', function() {
+          expect(metaTags[0].getAttribute('charset')).to.be.equal('utf-8');
+        });
+
+        it('should have default viewport <meta> tag', function() {
+          const viewportMeta = metaTags[1];
+          
+          expect(viewportMeta.getAttribute('name')).to.be.equal('viewport');
+          expect(viewportMeta.getAttribute('content')).to.be.equal('width=device-width, initial-scale=1');
+        });
+
+        it('should have default mobile-web-app-capable <meta> tag', function() {
+          const mwacMeta = metaTags[2];
+
+          expect(mwacMeta.getAttribute('name')).to.be.equal('mobile-web-app-capable');
+          expect(mwacMeta.getAttribute('content')).to.be.equal('yes');
+        });
+
+        it('should have default apple-mobile-web-app-capable <meta> tag', function() {
+          const amwacMeta = metaTags[3];
+
+          expect(amwacMeta.getAttribute('name')).to.be.equal('apple-mobile-web-app-capable');
+          expect(amwacMeta.getAttribute('content')).to.be.equal('yes');
+        });
+
+        it('should have default apple-mobile-web-app-status-bar-style <meta> tag', function() {
+          const amwasbsMeta = metaTags[4];
+
+          expect(amwasbsMeta.getAttribute('name')).to.be.equal('apple-mobile-web-app-status-bar-style');
+          expect(amwasbsMeta.getAttribute('content')).to.be.equal('black');
+        });
+      });
+
+      describe('content in the <body>', function() {
+        it('should have the expected content within the <app-header> tag', function() {
+          const heading = dom.window.document.querySelector('body app-header header').textContent;
+
+          expect(heading).to.equal('This is the header component.');
+        });
+
+        it('should have expected h2 tag in the <body>', function() {
+          const heading = dom.window.document.querySelector('body h2').textContent;
+    
+          expect(heading).to.be.equal('hello world');
+        });
+      });
+    });
+  });
+
+  after(function() {
+    setup.teardownTestBed();
+  });
+});
\ No newline at end of file
diff --git a/packages/cli/test/cases/build.default.quick-start-npx/src/components/header.js b/packages/cli/test/cases/build.default.quick-start-npx/src/components/header.js
new file mode 100644
index 00000000..61ee4639
--- /dev/null
+++ b/packages/cli/test/cases/build.default.quick-start-npx/src/components/header.js
@@ -0,0 +1,19 @@
+class HeaderComponent extends HTMLElement {
+  constructor() {
+    super();
+
+    this.root = this.attachShadow({ mode: 'open' });
+  }
+
+  connectedCallback() {
+    this.root.innerHTML = this.getTemplate();
+  }
+
+  getTemplate() {
+    return `
+      <header>This is the header component.</header>
+    `;
+  }
+}
+
+customElements.define('app-header', HeaderComponent);
\ No newline at end of file
diff --git a/packages/cli/test/cases/build.default.quick-start-npx/src/pages/index.md b/packages/cli/test/cases/build.default.quick-start-npx/src/pages/index.md
new file mode 100644
index 00000000..a2726c5c
--- /dev/null
+++ b/packages/cli/test/cases/build.default.quick-start-npx/src/pages/index.md
@@ -0,0 +1,1 @@
+## hello world
\ No newline at end of file
diff --git a/packages/cli/test/cases/build.default.quick-start-npx/src/templates/page.html b/packages/cli/test/cases/build.default.quick-start-npx/src/templates/page.html
new file mode 100644
index 00000000..2bc9053c
--- /dev/null
+++ b/packages/cli/test/cases/build.default.quick-start-npx/src/templates/page.html
@@ -0,0 +1,17 @@
+<!DOCTYPE html>
+<html lang="en" prefix="og:http://ogp.me/ns#">
+
+  <head>
+    <script type="module" src="/components/header.js"></script>
+  </head>
+
+  <body>
+
+    <div>
+      <app-header></app-header>
+      <content-outlet></content-outlet>
+    </div>
+
+  </body>
+
+</html>
\ No newline at end of file
diff --git a/test/test-bed.js b/test/test-bed.js
index 3eb24c85..60ae8db0 100644
--- a/test/test-bed.js
+++ b/test/test-bed.js
@@ -26,16 +26,19 @@ module.exports = class TestBed {
     this.enableStdOut = enableStdOut; // debugging tests
   }
 
-  setupTestBed(cwd, testFiles = []) {
+  setupTestBed(cwd, testFiles = [], useSetupFiles = true) {
     return new Promise(async (resolve, reject) => {
       try {
         this.rootDir = cwd;
         this.publicDir = path.join(this.rootDir, 'public');
         this.buildDir = path.join(this.rootDir, '.greenwood');
+        const allSetupFiles = useSetupFiles 
+          ? setupFiles.concat(testFiles)
+          : [].concat(testFiles);
 
         await this.teardownTestBed();
 
-        await Promise.all(setupFiles.concat(testFiles).map((file) => {
+        await Promise.all(allSetupFiles.map((file) => {
           return new Promise(async (resolve, reject) => {
             try {
               const targetSrc = path.join(process.cwd(), file.dir, file.name);