Repository: serverless/serverless (JavaScript)
Date: 2019-07-04 07:44:00+00:00
Task type: Feature

Issue:
Support enterprise Github with 'serverless install'
<!--
1. If you have a question and not a bug/feature request please ask it at http://forum.serverless.com
2. Please check if an issue already exists so there are no duplicates
3. Check out and follow our Guidelines: https://github.com/serverless/serverless/blob/master/CONTRIBUTING.md
4. Fill out the whole template so we have a good overview on the issue
5. Do not remove any section of the template. If something is not applicable leave it empty but leave it in the Issue
6. Please follow the template, otherwise we'll have to ask you to update it
-->

# This is a Feature Proposal

## Description
Right now the command `serverless install` statically checks if the url being passed is from `github.com`, `bitbucket.com`, or `gitlab.com`.  This means that it is impossible to run `serverless install` for an Enterprise Github repo.  `serverless install` should be expanded to support more potential domains, or a smarter validation should be done than just checking the hostname of the url.

[Code in question: ](https://github.com/serverless/serverless/blob/master/lib/utils/downloadTemplateFromRepo.js#L155-L169)

## Additional Data

* ***Serverless Framework Version you're using***: 1.28.0
* ***Operating System***: Mac OSX
* ***Stack Trace***: `none`
* ***Provider Error messages***: 

```
sls install --url https://<redacted_enterprise_github_url>/<org>/<repo>

  Serverless Error ---------------------------------------

  The URL you passed is not one of the valid providers: "GitHub", "BitBucket".
```



Issue discussion:
Given that `parseGitHubURL`, `parseBitbucketURL`, and `parseGitlabURL` are identical functions, it seems that only one parse function is needed. Most, if not all, will parse the same way, including GitHub Enterprise. 

I think a better validation would be to check if a git origin exists at the remote address:
```
GIT_TERMINAL_PROMPT=0 git ls-remote https://<origin_server>/<org>/<repo>
```

As the original poster stated, the current state makes it impossible to certain functionalities in an enterprise environment, including `sls install` and `sls create`, and this impacts using Serverless Framework templates.
This should be made possible by #5715 which was released in v1.36.3
Oops, misread. this is a github server other than github.com, not private repos 🤦‍♂️ 
Hello,
I'm running into this error after setting up some templates that i wanted to use from our github entreprise setup.

`sls create --template-url https://githubentreprise.mydomain.com/serverless-template/serverless/tree/master/non-prod/aws-nodejs --path test`

The above command returns:

`The URL you passed is not one of the valid providers: "GitHub", "Bitbucket", or "GitLab".`

I'm using sls version 1.43.0

Any information on how / when this can be possible? Can i help somehow?

/kalote

Solution patch:
diff --git a/docs/providers/aws/cli-reference/install.md b/docs/providers/aws/cli-reference/install.md
index 2220121e1ca..167dab1abe2 100644
--- a/docs/providers/aws/cli-reference/install.md
+++ b/docs/providers/aws/cli-reference/install.md
@@ -29,6 +29,13 @@ serverless install --url https://github.com/some/service
 
 - `install:install`
 
+## Supported Code Hosting Platforms
+
+- GitHub
+- GitHub Enterprise
+- GitLab
+- BitBucket
+
 ## Examples
 
 ### Installing a service from a GitHub URL
diff --git a/docs/providers/azure/cli-reference/install.md b/docs/providers/azure/cli-reference/install.md
index 805b24b2de5..a3c39473fdb 100644
--- a/docs/providers/azure/cli-reference/install.md
+++ b/docs/providers/azure/cli-reference/install.md
@@ -29,6 +29,13 @@ serverless install --url https://github.com/some/service
 
 - `install:install`
 
+## Supported Code Hosting Platforms
+
+- GitHub
+- GitHub Enterprise
+- GitLab
+- BitBucket
+
 ## Examples
 
 ### Installing a service from a GitHub URL
diff --git a/docs/providers/google/cli-reference/install.md b/docs/providers/google/cli-reference/install.md
index 4a2cfda3cc5..0406ffa7534 100644
--- a/docs/providers/google/cli-reference/install.md
+++ b/docs/providers/google/cli-reference/install.md
@@ -25,6 +25,13 @@ serverless install --url https://github.com/some/service
 - `--url` or `-u` The services GitHub URL. **Required**.
 - `--name` or `-n` Name for the service.
 
+## Supported Code Hosting Platforms
+
+- GitHub
+- GitHub Enterprise
+- GitLab
+- BitBucket
+
 ## Examples
 
 ### Installing a service from a GitHub URL
diff --git a/docs/providers/openwhisk/cli-reference/install.md b/docs/providers/openwhisk/cli-reference/install.md
index a43a74fe949..88a2ff60743 100644
--- a/docs/providers/openwhisk/cli-reference/install.md
+++ b/docs/providers/openwhisk/cli-reference/install.md
@@ -29,6 +29,13 @@ serverless install --url https://github.com/some/service
 
 - `install:install`
 
+## Supported Code Hosting Platforms
+
+- GitHub
+- GitHub Enterprise
+- GitLab
+- BitBucket
+
 ## Examples
 
 ### Installing a service from a GitHub URL
diff --git a/lib/utils/downloadTemplateFromRepo.js b/lib/utils/downloadTemplateFromRepo.js
index 49eaa7fe1be..1bbd0424b45 100644
--- a/lib/utils/downloadTemplateFromRepo.js
+++ b/lib/utils/downloadTemplateFromRepo.js
@@ -56,13 +56,16 @@ function parseGitHubURL(url) {
   const owner = parts[1];
   const repo = parts[2];
   const branch = isSubdirectory ? parts[pathLength] : 'master';
+  const isGitHubEnterprise = url.hostname !== 'github.com';
 
-  // validate if given url is a valid GitHub url
-  validateUrl(url, 'github.com', 'GitHub', owner, repo);
+  if (!isGitHubEnterprise) {
+    // validate if given url is a valid GitHub url
+    validateUrl(url, 'github.com', 'GitHub', owner, repo);
+  }
 
   const downloadUrl = `https://${
-    url.auth ? `${url.auth}@` : ''
-  }github.com/${owner}/${repo}/archive/${branch}.zip`;
+    isGitHubEnterprise ? url.hostname : 'github.com'
+  }/${owner}/${repo}/archive/${branch}.zip`;
 
   return {
     owner,
@@ -71,6 +74,7 @@ function parseGitHubURL(url) {
     downloadUrl,
     isSubdirectory,
     pathToDirectory: getPathDirectory(pathLength + 1, parts),
+    auth: url.auth || '',
   };
 }
 
@@ -100,6 +104,7 @@ function parseBitbucketURL(url) {
     downloadUrl,
     isSubdirectory,
     pathToDirectory: getPathDirectory(pathLength + 1, parts),
+    auth: '',
   };
 }
 
@@ -128,6 +133,7 @@ function parseGitlabURL(url) {
     downloadUrl,
     isSubdirectory,
     pathToDirectory: getPathDirectory(pathLength + 1, parts),
+    auth: '',
   };
 }
 
@@ -150,22 +156,17 @@ function parseRepoURL(inputUrl) {
     throw new ServerlessError('The URL you passed is not a valid URL');
   }
 
-  switch (url.hostname) {
-    case 'github.com': {
-      return parseGitHubURL(url);
-    }
-    case 'bitbucket.org': {
-      return parseBitbucketURL(url);
-    }
-    case 'gitlab.com': {
-      return parseGitlabURL(url);
-    }
-    default: {
-      const msg =
-        'The URL you passed is not one of the valid providers: "GitHub", "Bitbucket", or "GitLab".';
-      throw new ServerlessError(msg);
-    }
+  if (url.hostname === 'github.com' || url.hostname.indexOf('github.') !== -1) {
+    return parseGitHubURL(url);
+  } else if (url.hostname === 'bitbucket.org') {
+    return parseBitbucketURL(url);
+  } else if (url.hostname === 'gitlab.com') {
+    return parseGitlabURL(url);
   }
+
+  const msg =
+    'The URL you passed is not one of the valid providers: "GitHub", "GitHub Entreprise", "Bitbucket", or "GitLab".';
+  throw new ServerlessError(msg);
 }
 
 /**
@@ -180,6 +181,7 @@ function downloadTemplateFromRepo(inputUrl, templateName, downloadPath) {
   let serviceName;
   let dirName;
   let downloadServicePath;
+  const { auth } = repoInformation;
 
   if (repoInformation.isSubdirectory) {
     const folderName = repoInformation.pathToDirectory.split('/').splice(-1)[0];
@@ -202,13 +204,15 @@ function downloadTemplateFromRepo(inputUrl, templateName, downloadPath) {
 
   log(`Downloading and installing "${serviceName}"...`);
 
-  // download service
-  return download(repoInformation.downloadUrl, downloadServicePath, {
+  const downloadOptions = {
     timeout: 30000,
     extract: true,
     strip: 1,
     mode: '755',
-  })
+    auth,
+  };
+  // download service
+  return download(repoInformation.downloadUrl, downloadServicePath, downloadOptions)
     .then(() => {
       // if it's a directory inside of git
       if (repoInformation.isSubdirectory) {


Test patch:
diff --git a/lib/utils/downloadTemplateFromRepo.test.js b/lib/utils/downloadTemplateFromRepo.test.js
index b035d375afc..eee3f4c1a04 100644
--- a/lib/utils/downloadTemplateFromRepo.test.js
+++ b/lib/utils/downloadTemplateFromRepo.test.js
@@ -51,7 +51,7 @@ describe('downloadTemplateFromRepo', () => {
     });
 
     it('should throw an error if the passed URL is not a valid GitHub URL', () => {
-      expect(() => downloadTemplateFromRepo('http://no-github-url.com/foo/bar')).to.throw(Error);
+      expect(() => downloadTemplateFromRepo('http://no-git-hub-url.com/foo/bar')).to.throw(Error);
     });
 
     it('should throw an error if a directory with the same service name is already present', () => {
@@ -154,6 +154,7 @@ describe('downloadTemplateFromRepo', () => {
         downloadUrl: 'https://github.com/serverless/serverless/archive/master.zip',
         isSubdirectory: false,
         pathToDirectory: '',
+        auth: '',
       });
     });
 
@@ -167,6 +168,51 @@ describe('downloadTemplateFromRepo', () => {
         downloadUrl: 'https://github.com/serverless/serverless/archive/master.zip',
         isSubdirectory: true,
         pathToDirectory: 'assets',
+        auth: '',
+      });
+    });
+
+    it('should parse a valid GitHub Entreprise URL', () => {
+      const output = parseRepoURL('https://github.mydomain.com/serverless/serverless');
+
+      expect(output).to.deep.eq({
+        owner: 'serverless',
+        repo: 'serverless',
+        branch: 'master',
+        downloadUrl: 'https://github.mydomain.com/serverless/serverless/archive/master.zip',
+        isSubdirectory: false,
+        pathToDirectory: '',
+        auth: '',
+      });
+    });
+
+    it('should parse a valid GitHub Entreprise with subdirectory', () => {
+      const output = parseRepoURL(
+        'https://github.mydomain.com/serverless/serverless/tree/master/assets'
+      );
+
+      expect(output).to.deep.eq({
+        owner: 'serverless',
+        repo: 'serverless',
+        branch: 'master',
+        downloadUrl: 'https://github.mydomain.com/serverless/serverless/archive/master.zip',
+        isSubdirectory: true,
+        pathToDirectory: 'assets',
+        auth: '',
+      });
+    });
+
+    it('should parse a valid GitHub Entreprise URL with authentication', () => {
+      const output = parseRepoURL('https://username:password@github.com/serverless/serverless/');
+
+      expect(output).to.deep.eq({
+        owner: 'serverless',
+        repo: 'serverless',
+        branch: 'master',
+        downloadUrl: 'https://github.com/serverless/serverless/archive/master.zip',
+        isSubdirectory: false,
+        auth: 'username:password',
+        pathToDirectory: '',
       });
     });
 
@@ -180,6 +226,7 @@ describe('downloadTemplateFromRepo', () => {
         downloadUrl: 'https://bitbucket.org/atlassian/localstack/get/master.zip',
         isSubdirectory: false,
         pathToDirectory: '',
+        auth: '',
       });
     });
 
@@ -195,6 +242,7 @@ describe('downloadTemplateFromRepo', () => {
         downloadUrl: 'https://bitbucket.org/atlassian/localstack/get/mvn.zip',
         isSubdirectory: true,
         pathToDirectory: `localstack${path.sep}dashboard`,
+        auth: '',
       });
     });
 
@@ -209,6 +257,7 @@ describe('downloadTemplateFromRepo', () => {
           'https://gitlab.com/serverless/serverless/-/archive/master/serverless-master.zip',
         isSubdirectory: false,
         pathToDirectory: '',
+        auth: '',
       });
     });
 
@@ -222,6 +271,7 @@ describe('downloadTemplateFromRepo', () => {
         downloadUrl: 'https://gitlab.com/serverless/serverless/-/archive/dev/serverless-dev.zip',
         isSubdirectory: true,
         pathToDirectory: 'subdir',
+        auth: '',
       });
     });
   });