Repository: serverless/serverless (JavaScript) — Feature

Issue:
Unable to download template from a private repo or an internal path
# This is a Feature Proposal

## Description

* When doing a `serverless create`, we can specify a `template-url` to fetch the template from a git repo. However this does not works with private repositories because it only use HTTPS and not SSH. There is no option to specify a local path for the template.
* We could add a new option `--template-path` that would avoid downloading the template from a repo by copying files from the local path. Let me know if that makes sense, I could create a PR to add this feature.

## Additional Data

* ***Serverless Framework Version you're using***: 8.8.0
* ***Operating System***: darwin
* ***Stack Trace***: `HTTPError: Response code 404 (Not Found)`


Discussion:
Hi @ChristopheBougere ,

thanks for the feature request. So you have the original use case where you wanted to fetch a template via a git ssh url (`ssh://git@....`) and the follow up use case where you want to read a template source from a local directory?

For me both use cases make sense (the first one in general in a company, the second one rather for a developer working local). So both of them should be covered in this feature request.

Resolving patch:
diff --git a/docs/providers/aws/cli-reference/create.md b/docs/providers/aws/cli-reference/create.md
index fc38bf9cd8d..73421f72533 100644
--- a/docs/providers/aws/cli-reference/create.md
+++ b/docs/providers/aws/cli-reference/create.md
@@ -33,8 +33,9 @@ serverless create --template-url https://github.com/serverless/serverless/tree/m
 ```
 
 ## Options
-- `--template` or `-t` The name of one of the available templates. **Required if --template-url is not present**.
-- `--template-url` or `-u` The name of one of the available templates. **Required if --template is not present**.
+- `--template` or `-t` The name of one of the available templates. **Required if --template-url and --template-path are not present**.
+- `--template-url` or `-u` The name of one of the available templates. **Required if --template and --template-path are not present**.
+- `--template-path` The local path of your template. **Required if --template and --template-url are not present**.
 - `--path` or `-p` The path where the service should be created.
 - `--name` or `-n` the name of the service in `serverless.yml`.
 
@@ -90,6 +91,14 @@ will use the already present directory.
 Additionally Serverless will rename the service according to the path you provide. In this example the service will be
 renamed to `my-new-service`.
 
+### Creating a new service using a local template
+
+```bash
+serverless create --template-path path/to/my/template/folder --path path/to/my/service --name my-new-service
+```
+
+This will copy the `path/to/my/template/folder` folder into `path/to/my/service` and rename the service to `my-new-service`.
+
 ### Creating a new plugin
 
 ```
diff --git a/docs/providers/azure/cli-reference/create.md b/docs/providers/azure/cli-reference/create.md
index 998a09cef3e..f9bddeeaa54 100644
--- a/docs/providers/azure/cli-reference/create.md
+++ b/docs/providers/azure/cli-reference/create.md
@@ -28,8 +28,9 @@ serverless create --template azure-nodejs --path myService
 ```
 
 ## Options
-- `--template` or `-t` The name of one of the available templates. **Required if --template-url is not present**.
-- `--template-url` or `-u` The name of one of the available templates. **Required if --template is not present**.
+- `--template` or `-t` The name of one of the available templates. **Required if --template-url and --template-path are not present**.
+- `--template-url` or `-u` The name of one of the available templates. **Required if --template and --template-path are not present**.
+- `--template-path` The local path of your template. **Required if --template and --template-url are not present**.
 - `--path` or `-p` The path where the service should be created.
 - `--name` or `-n` the name of the service in `serverless.yml`.
 
@@ -70,6 +71,14 @@ Serverless will use the already present directory.
 Additionally Serverless will rename the service according to the path you
 provide. In this example the service will be renamed to `my-new-service`.
 
+### Creating a new service using a local template
+
+```bash
+serverless create --template-path path/to/my/template/folder --path path/to/my/service --name my-new-service
+```
+
+This will copy the `path/to/my/template/folder` folder into `path/to/my/service` and rename the service to `my-new-service`.
+
 ### Create service in new folder using a custom template
 
 ```bash
diff --git a/docs/providers/google/cli-reference/create.md b/docs/providers/google/cli-reference/create.md
index 232c84bdc08..27d550bdb33 100644
--- a/docs/providers/google/cli-reference/create.md
+++ b/docs/providers/google/cli-reference/create.md
@@ -28,8 +28,9 @@ serverless create --template google-nodejs --path my-service
 
 ## Options
 
-- `--template` or `-t` The name of one of the available templates. **Required if --template-url is not present**.
-- `--template-url` or `-u` The name of one of the available templates. **Required if --template is not present**.
+- `--template` or `-t` The name of one of the available templates. **Required if --template-url and --template-path are not present**.
+- `--template-url` or `-u` The name of one of the available templates. **Required if --template and --template-path are not present**.
+- `--template-path` The local path of your template. **Required if --template and --template-url are not present**.
 - `--path` or `-p` The path where the service should be created.
 - `--name` or `-n` the name of the service in `serverless.yml`.
 
@@ -61,6 +62,14 @@ This example will generate scaffolding for a service with `google` as a provider
 
 Additionally Serverless will rename the service according to the path you provide. In this example the service will be renamed to `my-new-service`.
 
+### Creating a new service using a local template
+
+```bash
+serverless create --template-path path/to/my/template/folder --path path/to/my/service --name my-new-service
+```
+
+This will copy the `path/to/my/template/folder` folder into `path/to/my/service` and rename the service to `my-new-service`.
+
 ### Create service in new folder using a custom template
 
 ```bash
diff --git a/docs/providers/kubeless/cli-reference/create.md b/docs/providers/kubeless/cli-reference/create.md
index 7911ef2810c..68a1bd6b3b2 100644
--- a/docs/providers/kubeless/cli-reference/create.md
+++ b/docs/providers/kubeless/cli-reference/create.md
@@ -35,8 +35,9 @@ serverless create --template kubeless-nodejs --path my-service
 ```
 
 ## Options
-- `--template` or `-t` The name of one of the available templates. **Required if --template-url is not present**.
-- `--template-url` or `-u` The name of one of the available templates. **Required if --template is not present**.
+- `--template` or `-t` The name of one of the available templates. **Required if --template-url and --template-path are not present**.
+- `--template-url` or `-u` The name of one of the available templates. **Required if --template and --template-path are not present**.
+- `--template-path` The local path of your template. **Required if --template and --template-url are not present**.
 - `--path` or `-p` The path where the service should be created.
 - `--name` or `-n` the name of the service in `serverless.yml`.
 
@@ -73,3 +74,11 @@ serverless create --template kubeless-python --path my-new-service
 This example will generate scaffolding for a service with `kubeless` as a provider and `python2.7` as runtime. The scaffolding will be generated in the `my-new-service` directory. This directory will be created if not present. Otherwise Serverless will use the already present directory.
 
 Additionally Serverless will rename the service according to the path you provide. In this example the service will be renamed to `my-new-service`.
+
+### Creating a new service using a local template
+
+```bash
+serverless create --template-path path/to/my/template/folder --path path/to/my/service --name my-new-service
+```
+
+This will copy the `path/to/my/template/folder` folder into `path/to/my/service` and rename the service to `my-new-service`.
diff --git a/docs/providers/openwhisk/cli-reference/create.md b/docs/providers/openwhisk/cli-reference/create.md
index 5c5201e47b6..952029e9652 100644
--- a/docs/providers/openwhisk/cli-reference/create.md
+++ b/docs/providers/openwhisk/cli-reference/create.md
@@ -27,8 +27,9 @@ serverless create --template openwhisk-nodejs --path myService
 ```
 
 ## Options
-- `--template` or `-t` The name of one of the available templates. **Required if --template-url is not present**.
-- `--template-url` or `-u` The name of one of the available templates. **Required if --template is not present**.
+- `--template` or `-t` The name of one of the available templates. **Required if --template-url and --template-path are not present**.
+- `--template-url` or `-u` The name of one of the available templates. **Required if --template and --template-path are not present**.
+- `--template-path` The local path of your template. **Required if --template and --template-url are not present**.
 - `--path` or `-p` The path where the service should be created.
 - `--name` or `-n` the name of the service in `serverless.yml`.
 
@@ -69,6 +70,14 @@ This example will generate scaffolding for a service with `openwhisk` as a provi
 
 Additionally Serverless will rename the service according to the path you provide. In this example the service will be renamed to `my-new-service`.
 
+### Creating a new service using a local template
+
+```bash
+serverless create --template-path path/to/my/template/folder --path path/to/my/service --name my-new-service
+```
+
+This will copy the `path/to/my/template/folder` folder into `path/to/my/service` and rename the service to `my-new-service`.
+
 ### Create service in new folder using a custom template
 
 ```bash
diff --git a/docs/providers/spotinst/cli-reference/create.md b/docs/providers/spotinst/cli-reference/create.md
index 0009dea57c5..fa4421d7e60 100755
--- a/docs/providers/spotinst/cli-reference/create.md
+++ b/docs/providers/spotinst/cli-reference/create.md
@@ -27,8 +27,9 @@ serverless create -t spotinst-nodejs -p myService
 ```
 
 ## Options
-- `--template` or `-t` The name of one of the available templates. **Required if --template-url is not present**.
-- `--template-url` or `-u` The name of one of the available templates. **Required if --template is not present**.
+- `--template` or `-t` The name of one of the available templates. **Required if --template-url and --template-path are not present**.
+- `--template-url` or `-u` The name of one of the available templates. **Required if --template and --template-path are not present**.
+- `--template-path` The local path of your template. **Required if --template and --template-url are not present**.
 - `--path` or `-p` The path where the service should be created.
 - `--name` or `-n` the name of the service in `serverless.yml`.
 
@@ -70,3 +71,11 @@ serverless create -t spotinst-nodejs -p my-new-service
 This example will generate scaffolding for a service with `Spotinst` as a provider and `ruby` as runtime. The scaffolding
 will be generated in the `my-new-service` directory. This directory will be created if not present. Otherwise Serverless
 will use the already present directory.
+
+### Creating a new service using a local template
+
+```bash
+serverless create --template-path path/to/my/template/folder --path path/to/my/service --name my-new-service
+```
+
+This will copy the `path/to/my/template/folder` folder into `path/to/my/service` and rename the service to `my-new-service`.
diff --git a/docs/providers/webtasks/cli-reference/create.md b/docs/providers/webtasks/cli-reference/create.md
index 56da10709cf..779f57738d2 100755
--- a/docs/providers/webtasks/cli-reference/create.md
+++ b/docs/providers/webtasks/cli-reference/create.md
@@ -28,8 +28,9 @@ serverless create --template webtasks-nodejs --path my-service
 
 ## Options
 
-- `--template` or `-t` The name of one of the available templates. **Required if --template-url is not present**.
-- `--template-url` or `-u` The name of one of the available templates. **Required if --template is not present**.
+- `--template` or `-t` The name of one of the available templates. **Required if --template-url and --template-path are not present**.
+- `--template-url` or `-u` The name of one of the available templates. **Required if --template and --template-path are not present**.
+- `--template-path` The local path of your template. **Required if --template and --template-url are not present**.
 - `--path` or `-p` The path where the service should be created.
 - `--name` or `-n` the name of the service in `serverless.yml`.
 
@@ -60,3 +61,11 @@ serverless create --template webtasks-nodejs --path my-new-service
 This example will generate scaffolding for a service with `webtasks` as a provider. The scaffolding will be generated in the `my-new-service` directory. This directory will be created if not present. Otherwise Serverless will use the already present directory.
 
 Additionally Serverless will rename the service according to the path you provide. In this example the service will be renamed to `my-new-service`.
+
+### Creating a new service using a local template
+
+```bash
+serverless create --template-path path/to/my/template/folder --path path/to/my/service --name my-new-service
+```
+
+This will copy the `path/to/my/template/folder` folder into `path/to/my/service` and rename the service to `my-new-service`.
diff --git a/lib/plugins/create/create.js b/lib/plugins/create/create.js
index eaa3a67833f..471c9167099 100644
--- a/lib/plugins/create/create.js
+++ b/lib/plugins/create/create.js
@@ -5,8 +5,12 @@ const path = require('path');
 const fse = require('fs-extra');
 const _ = require('lodash');
 
+const ServerlessError = require('../../classes/Error').ServerlessError;
 const userStats = require('../../utils/userStats');
 const download = require('../../utils/downloadTemplateFromRepo');
+const renameService = require('../../utils/renameService').renameService;
+const copyDirContentsSync = require('../../utils/fs/copyDirContentsSync');
+const dirExistsSync = require('../../utils/fs/dirExistsSync');
 
 // class wide constants
 const validTemplates = [
@@ -66,6 +70,9 @@ class Create {
             usage: 'Template URL for the service. Supports: GitHub, BitBucket',
             shortcut: 'u',
           },
+          'template-path': {
+            usage: 'Template local path for the service.',
+          },
           path: {
             usage: 'The path where the service should be created (e.g. --path my-service)',
             shortcut: 'p',
@@ -111,10 +118,23 @@ class Create {
         .catch(err => {
           throw new this.serverless.classes.Error(err);
         });
+    } else if ('template-path' in this.options) {
+      // Copying template from a local directory
+      const servicePath = this.options.path || path.join(process.cwd(), this.options.name);
+      if (dirExistsSync(servicePath)) {
+        const errorMessage = `A folder named "${servicePath}" already exists.`;
+        throw new ServerlessError(errorMessage);
+      }
+      copyDirContentsSync(this.options['template-path'], servicePath, {
+        noLinks: true,
+      });
+      if (this.options.name) {
+        renameService(this.options.name, servicePath);
+      }
     } else {
       const errorMessage = [
-        'You must either pass a template name (--template) or a ',
-        'a URL (--template-url).',
+        'You must either pass a template name (--template), ',
+        'a URL (--template-url) or a local path (--template-path).',
       ].join('');
       throw new this.serverless.classes.Error(errorMessage);
     }
diff --git a/lib/utils/fs/copyDirContentsSync.js b/lib/utils/fs/copyDirContentsSync.js
index 5c38378b264..dc75de7a653 100644
--- a/lib/utils/fs/copyDirContentsSync.js
+++ b/lib/utils/fs/copyDirContentsSync.js
@@ -4,8 +4,8 @@ const path = require('path');
 const fse = require('./fse');
 const walkDirSync = require('./walkDirSync');
 
-function fileExists(srcDir, destDir) {
-  const fullFilesPaths = walkDirSync(srcDir);
+function fileExists(srcDir, destDir, options) {
+  const fullFilesPaths = walkDirSync(srcDir, options);
 
   fullFilesPaths.forEach(fullFilePath => {
     const relativeFilePath = fullFilePath.replace(srcDir, '');
diff --git a/lib/utils/fs/walkDirSync.js b/lib/utils/fs/walkDirSync.js
index 0e4615272b1..150e856b065 100644
--- a/lib/utils/fs/walkDirSync.js
+++ b/lib/utils/fs/walkDirSync.js
@@ -3,15 +3,21 @@
 const path = require('path');
 const fs = require('fs');
 
-function walkDirSync(dirPath) {
+function walkDirSync(dirPath, opts) {
+  const options = Object.assign({
+    noLinks: false,
+  }, opts);
   let filePaths = [];
   const list = fs.readdirSync(dirPath);
   list.forEach((filePathParam) => {
     let filePath = filePathParam;
     filePath = path.join(dirPath, filePath);
-    const stat = fs.statSync(filePath);
-    if (stat && stat.isDirectory()) {
-      filePaths = filePaths.concat(walkDirSync(filePath));
+    const stat = options.noLinks ? fs.lstatSync(filePath) : fs.statSync(filePath);
+    // skipping symbolic links when noLinks option
+    if (options.noLinks && stat && stat.isSymbolicLink()) {
+      return;
+    } else if (stat && stat.isDirectory()) {
+      filePaths = filePaths.concat(walkDirSync(filePath, opts));
     } else {
       filePaths.push(filePath);
     }


Test patch:
diff --git a/lib/plugins/create/create.test.js b/lib/plugins/create/create.test.js
index df578b49347..531d4fd7159 100644
--- a/lib/plugins/create/create.test.js
+++ b/lib/plugins/create/create.test.js
@@ -666,5 +666,48 @@ describe('Create', () => {
 
       return create.create().catch(() => download.downloadTemplateFromRepo.restore());
     });
+
+    it('should copy "aws-nodejs" template from local path', () => {
+      process.chdir(tmpDir);
+      const distDir = path.join(tmpDir, 'my-awesome-service');
+      create.options = {};
+      create.options.path = distDir;
+      create.options['template-path'] = path.join(__dirname, 'templates/aws-nodejs');
+
+      return create.create().then(() => {
+        const dirContent = fs.readdirSync(distDir);
+
+        expect(dirContent).to.include('serverless.yml');
+        expect(dirContent).to.include('handler.js');
+        expect(dirContent).to.include('gitignore');
+
+        // check if the service was renamed
+        const serverlessYmlfileContent = fse
+          .readFileSync(path.join(distDir, 'serverless.yml')).toString();
+
+        expect((/service: aws-nodejs/).test(serverlessYmlfileContent)).to.equal(true);
+      });
+    });
+
+    it('should copy "aws-nodejs" template from local path with a custom name', () => {
+      process.chdir(tmpDir);
+      create.options = {};
+      create.options['template-path'] = path.join(__dirname, 'templates/aws-nodejs');
+      create.options.name = 'my-awesome-service';
+
+      return create.create().then(() => {
+        const dirContent = fs.readdirSync(path.join(tmpDir, 'my-awesome-service'));
+
+        expect(dirContent).to.include('serverless.yml');
+        expect(dirContent).to.include('handler.js');
+        expect(dirContent).to.include('gitignore');
+
+        // check if the service was renamed
+        const serverlessYmlfileContent = fse
+          .readFileSync(path.join(tmpDir, 'my-awesome-service', 'serverless.yml')).toString();
+
+        expect((/service: my-awesome-service/).test(serverlessYmlfileContent)).to.equal(true);
+      });
+    });
   });
 });
diff --git a/lib/utils/fs/walkDirSync.test.js b/lib/utils/fs/walkDirSync.test.js
index 991e9d06db0..2ef3a2600a9 100644
--- a/lib/utils/fs/walkDirSync.test.js
+++ b/lib/utils/fs/walkDirSync.test.js
@@ -1,5 +1,6 @@
 'use strict';
 
+const fs = require('fs');
 const path = require('path');
 const expect = require('chai').expect;
 const testUtils = require('../../../tests/utils');
@@ -28,4 +29,21 @@ describe('#walkDirSync()', () => {
     expect(filePaths).to.include(tmpFilePath2);
     expect(filePaths).to.include(tmpFilePath3);
   });
+
+  it('should check noLinks option', () => {
+    const tmpDirPath = testUtils.getTmpDirPath();
+
+    const realFile = path.join(tmpDirPath, 'real');
+    writeFileSync(realFile, 'content');
+
+    const symLink = path.join(tmpDirPath, 'sym');
+    fs.symlinkSync(realFile, symLink);
+
+    const filePaths = walkDirSync(tmpDirPath, {
+      noLinks: true,
+    });
+
+    expect(filePaths).to.include(realFile);
+    expect(filePaths).not.to.include(symLink);
+  });
 });