Repository: serverless/serverless (JavaScript)
Date: 2017-02-01 21:21:14+00:00
Task type: Feature

Issue:
Allow plugins to be loaded from parent node_modules folder
# This is a Feature Proposal

## Description

V.1 conflates dev dependencies and regular dependencies but gives you no easy way to exclude the dev dependencies, causing Lambdas to be larger than necessary.  

V.0 resolved this issue by having dev dependencies installed in the parent folder (formerly known as the project folder).  

By allowing plugins to be loaded from a parent `node_modules` folder, not just the current service folder, we can easily solve this issue since the service folder is the deployment unit.


Issue discussion:
What folder layout are you expecting with this @ac360? Eg below.
- Root folder
  - Service Folder
    - serverless.yml
  - node_modules

@ac360 wouldn't this be a problem for loading devDependencies like mocha as npm expects them to be in a local node_modules folder? So when I want to execute the mocha binary will it be found?

Just tested it by installing mocha in a parent npm folder (and looked into the docs) and it seems like npm is then only loading dependencies in the node_modules folder in the same directory?

To optimise your dependencies and remove dev dependencies you can also use `npm prune --production` which will remove all dev dependencies from your node_modules folder.

Or is this only for additional plugins for users? Because I think the impact of only having support for plugins in other node_modules folders will be pretty minimal.

@flomotlik npm automatically searches several parent folders (I think it's ~10) when it can't find a module in the local `node_modules` folder.  But the Framework doesn't follow this pattern and only looks [locally](https://github.com/serverless/serverless/blob/master/lib/classes/PluginManager.js#L81).  This can be changed with a few trivial lines of code.

One of the largest problems of the Serverless Framework is slow development time.  Speeding this up is critical to its success.  The majority of the slowness clearly comes from the upload time of the Lambda zip.  Therefore, reducing the size of the zip is most obvious way to speed up development, and this is why combining dev dependencies with regular dependencies is hugely problematic.

Running `npm prune --production` is one way, but it doesn't fix the underlying problem of slowness since the user will have to run this before each deploy, then run `install`, when they continue to develop.

Keeping dev dependencies in a parent folder isn't perfect, since we'd optimally want to declare those in the `package.json` in the service folder, so that everything is defined together.  However, it offers the best solution to the problem at this time and does so as an option, not a forced way or recommended best practice, allowing us more time to figure that out.

Are you sure it goes up? I tried this recently and if there was a node_modules folder in the same folder it didn’t look into the parent anymore. The other thing is that this will definitely require users to split up their dev/production dependencies and install them in different places. How would they do this and how many would like to do this? From my experience splitting up dependencies into different files and folders gets very painful very quickly so I’m not sure how practical this is and how many users would actually go for it.

And even then wouldn’t this only touch plugins they install from us? This only covers a small part of dev dependencies as you have a ll the test tools, … in place as well, so it would only provide a minimal upside and require quite a bit of work from the users.

On 13 November 2016 at 21:43:28, Austen (notifications@github.com) wrote:

@flomotlik npm automatically searches several parent folders (I think it's ~10) when it can't find a module in the local node_modules folder. But the Framework doesn't follow this pattern and only looks locally. This can be changed with a few trivial lines of code.

One of the largest problems of the Serverless Framework is slow development time. Speeding this up is critical to its success. The majority of the slowness clearly comes from the upload time of the Lambda zip. Therefore, reducing the size of the zip is most obvious way to speed up development, and this is why combining dev dependencies with regular dependencies is hugely problematic.

Running npm prune --production is one way, but it doesn't fix the underlying problem of slowness since the user will have to run this before each deploy, then run install, when they continue to develop.

Keeping dev dependencies in a parent folder isn't perfect, since we'd optimally want to declare those in the package.json in the service folder, so that everything is defined together. However, it offers the best solution to the problem at this time and does so as an option, not a forced way or recommended best practice, allowing us more time to figure that out.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.


Solution patch:
diff --git a/lib/classes/PluginManager.js b/lib/classes/PluginManager.js
index 4eda68d8f43..96f60ea9051 100644
--- a/lib/classes/PluginManager.js
+++ b/lib/classes/PluginManager.js
@@ -1,6 +1,7 @@
 'use strict';
 
 const path = require('path');
+const Module = require('module');
 const BbPromise = require('bluebird');
 const _ = require('lodash');
 
@@ -75,20 +76,15 @@ class PluginManager {
   loadServicePlugins(servicePlugs) {
     const servicePlugins = Array.isArray(servicePlugs) ? servicePlugs : [];
 
+    // eslint-disable-next-line no-underscore-dangle
+    module.paths = Module._nodeModulePaths(process.cwd());
+
     // we want to load plugins installed locally in the service
     if (this.serverless && this.serverless.config && this.serverless.config.servicePath) {
-      module.paths.unshift(
-        path.join(this.serverless.config.servicePath, 'node_modules'),
-        path.join(this.serverless.config.servicePath, '.serverless_plugins')
-        );
+      module.paths.unshift(path.join(this.serverless.config.servicePath, '.serverless_plugins'));
     }
 
     this.loadPlugins(servicePlugins);
-
-    // restore module paths
-    if (this.serverless && this.serverless.config && this.serverless.config.servicePath) {
-      module.paths.shift();
-    }
   }
 
   loadCommand(pluginName, details, key) {


Test patch:
diff --git a/lib/classes/PluginManager.test.js b/lib/classes/PluginManager.test.js
index b4155d3064e..9bb3a3dd8ef 100644
--- a/lib/classes/PluginManager.test.js
+++ b/lib/classes/PluginManager.test.js
@@ -6,6 +6,7 @@ const Serverless = require('../../lib/Serverless');
 const Create = require('../../lib/plugins/create/create');
 
 const path = require('path');
+const fs = require('fs');
 const fse = require('fs-extra');
 const execSync = require('child_process').execSync;
 const mockRequire = require('mock-require');
@@ -739,26 +740,50 @@ describe('PluginManager', () => {
   describe('Plugin / CLI integration', function () {
     this.timeout(0);
 
-    it('should expose a working integration between the CLI and the plugin system', () => {
-      const serverlessInstance = new Serverless();
+    const cwd = process.cwd();
+    let serverlessInstance;
+    let serviceDir;
+    let serverlessExec;
+
+    beforeEach(function () { // eslint-disable-line prefer-arrow-callback
+      serverlessInstance = new Serverless();
       serverlessInstance.init();
 
       // Cannot rely on shebang in severless.js to invoke script using NodeJS on Windows.
       const execPrefix = os.platform() === 'win32' ? 'node ' : '';
-      const serverlessExec = execPrefix + path.join(serverlessInstance.config.serverlessPath,
+      serverlessExec = execPrefix + path.join(serverlessInstance.config.serverlessPath,
               '..', 'bin', 'serverless');
       const tmpDir = testUtils.getTmpDirPath();
-      fse.mkdirsSync(tmpDir);
-      const cwd = process.cwd();
-      process.chdir(tmpDir);
+      serviceDir = path.join(tmpDir, 'service');
+      fse.mkdirsSync(serviceDir);
+      process.chdir(serviceDir);
 
       execSync(`${serverlessExec} create --template aws-nodejs`);
+    });
 
+    it('should expose a working integration between the CLI and the plugin system', () => {
       expect(serverlessInstance.utils
-        .fileExistsSync(path.join(tmpDir, 'serverless.yml'))).to.equal(true);
+        .fileExistsSync(path.join(serviceDir, 'serverless.yml'))).to.equal(true);
       expect(serverlessInstance.utils
-        .fileExistsSync(path.join(tmpDir, 'handler.js'))).to.equal(true);
+        .fileExistsSync(path.join(serviceDir, 'handler.js'))).to.equal(true);
+    });
 
+    it('should load plugins relatively to the working directory', () => {
+      const localPluginDir = path.join(serviceDir, 'node_modules', 'local-plugin');
+      const parentPluginDir = path.join(serviceDir, '..', 'node_modules', 'parent-plugin');
+      testUtils.installPlugin(localPluginDir, SynchronousPluginMock);
+      testUtils.installPlugin(parentPluginDir, PromisePluginMock);
+
+      fs.appendFileSync(path.join(serviceDir, 'serverless.yml'),
+        'plugins:\n  - local-plugin\n  - parent-plugin');
+
+      const output = execSync(serverlessExec);
+      const stringifiedOutput = (new Buffer(output, 'base64').toString());
+      expect(stringifiedOutput).to.contain('SynchronousPluginMock');
+      expect(stringifiedOutput).to.contain('PromisePluginMock');
+    });
+
+    afterEach(function () { // eslint-disable-line prefer-arrow-callback
       process.chdir(cwd);
     });
   });
diff --git a/tests/utils/index.js b/tests/utils/index.js
index 9b23eb8a56c..e9e8701ad62 100644
--- a/tests/utils/index.js
+++ b/tests/utils/index.js
@@ -84,6 +84,14 @@ module.exports = {
     return SNS.createTopicPromised(params);
   },
 
+  installPlugin: (installDir, PluginClass) => {
+    const pluginPkg = { name: path.basename(installDir), version: '0.0.0' };
+    const className = (new PluginClass()).constructor.name;
+    fse.outputFileSync(path.join(installDir, 'package.json'), JSON.stringify(pluginPkg), 'utf8');
+    fse.outputFileSync(path.join(installDir, 'index.js'),
+      `"use strict";\n${PluginClass.toString()}\nmodule.exports = ${className}`, 'utf8');
+  },
+
   removeSnsTopic(topicName) {
     const SNS = new AWS.SNS({ region: 'us-east-1' });
     BbPromise.promisifyAll(SNS, { suffix: 'Promised' });