Repository: serverless/serverless (JavaScript)
Date: 2019-06-12 17:13:27+00:00
Task type: Bug Fix

Issue:
Upload size of AWS go lambda's doubles with 1.36.xx
<!--
1. If you have a question and not a bug report please ask first at http://forum.serverless.com
2. Please check if an issue already exists. This bug may have already been documented
3. Check out and follow our Guidelines: https://github.com/serverless/serverless/blob/master/CONTRIBUTING.md
4. Fill out the whole template so we have a good overview on the issue
5. Do not remove any section of the template. If something is not applicable leave it empty but leave it in the Issue
6. Please follow the template, otherwise we'll have to ask you to update it
-->

# This is a Bug Report
I have noticed that the size of the zip files created by Serverless to upload Golang lambdas to AWS have doubled in size with the update of Serverless from 1.34.xx to 1.36.xxx, any reason for this or is it a bug?

## Description

* What went wrong?
* What did you expect should have happened?
* What was the config you used?
* What stacktrace or error message from your provider did you see?

Similar or dependent issues:
* #12345

## Additional Data

* ***Serverless Framework Version you're using***:
* ***Operating System***:
* ***Stack Trace***:
* ***Provider Error messages***:



Issue discussion:
it seems that the files in the zip files are not compressed
We've noticed the same issue. Basically, the files in .zip are duplicated;

```
Archive:  service-function.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
 35564224  1980-01-01 00:00   ./bin/api
 29804640  1980-01-01 00:00   ./bin/authorizer
 29706272  1980-01-01 00:00   ./bin/notify
 30120192  1980-01-01 00:00   ./bin/scan
 35564224  1980-01-01 00:00   bin/api
 29804640  1980-01-01 00:00   bin/authorizer
 29706272  1980-01-01 00:00   bin/notify
 30120192  1980-01-01 00:00   bin/scan
---------                     -------
250390656                     8 files
```
The service is written in Go. 

I have checked the zip produced for freshly created service with templates: aws-go-mod and aws-python3. For go, the zip produced by sls contains duplicated files, whereas for python there is no issue.

Here is the example freshly created from aws-go-mod template:
```
Archive:  service-demo-go-2.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
  6599296  1980-01-01 00:00   ./bin/hello
  6599296  1980-01-01 00:00   ./bin/world
  6599296  1980-01-01 00:00   bin/hello
  6599296  1980-01-01 00:00   bin/world
---------                     -------
 26397184                     4 files
```

and the one created for Python (added additional file with second handler to have two lambda functions):
```
Archive:  service-demo-2.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
      497  1980-01-01 00:00   handle2.py
      497  1980-01-01 00:00   handler.py
---------                     -------
      994                     2 files
```


My version of SLS and Node:
```
  Your Environment Information -----------------------------
     OS:                     linux
     Node Version:           8.15.0
     Serverless Version:     1.36.3
 
```

One more comment, it looks like the issue is due to the `package` section in `serverless.yml`. Namely, for aws-go the template contains following part:
```yaml
package:
 exclude:
   - ./**
 include:
   - ./bin/**
```
which I assume  causes that the handlers are copied two times. 
The first time because of the `include` parameter and the second time because of the `handler: bin/hello`
If you use
```
package:
 exclude:
   - ./**
 include:
   - bin/**
```
instead of
```
package:
 exclude:
   - ./**
 include:
   - ./bin/**
```
there will be no duplicated files


Solution patch:
diff --git a/lib/plugins/package/lib/packageService.js b/lib/plugins/package/lib/packageService.js
index ab6d27a82ed..22792f78a45 100644
--- a/lib/plugins/package/lib/packageService.js
+++ b/lib/plugins/package/lib/packageService.js
@@ -225,6 +225,7 @@ module.exports = {
     params.include.forEach((pattern) => {
       patterns.push(pattern);
     });
+
     // NOTE: please keep this order of concatenating the include params
     // rather than doing it the other way round!
     // see https://github.com/serverless/serverless/pull/5825 for more information
diff --git a/lib/plugins/package/lib/zipService.js b/lib/plugins/package/lib/zipService.js
index 9b7137c271d..2d53538bc40 100644
--- a/lib/plugins/package/lib/zipService.js
+++ b/lib/plugins/package/lib/zipService.js
@@ -82,27 +82,29 @@ module.exports = {
       output.on('error', (err) => reject(err));
       zip.on('error', (err) => reject(err));
 
-
       output.on('open', () => {
         zip.pipe(output);
 
-        BbPromise.all(files.map(this.getFileContentAndStat.bind(this))).then((contents) => {
-          _.forEach(_.sortBy(contents, ['filePath']), (file) => {
-            const name = file.filePath.slice(prefix ? `${prefix}${path.sep}`.length : 0);
-            let mode = file.stat.mode;
-            if (filesToChmodPlusX && _.includes(filesToChmodPlusX, name)
-                && file.stat.mode % 2 === 0) {
-              mode += 1;
-            }
-            zip.append(file.data, {
-              name,
-              mode,
-              date: new Date(0), // necessary to get the same hash when zipping the same content
+        const normalizedFiles = _.uniq(files.map(file => path.normalize(file)));
+
+        return BbPromise.all(normalizedFiles.map(this.getFileContentAndStat.bind(this)))
+          .then((contents) => {
+            _.forEach(_.sortBy(contents, ['filePath']), (file) => {
+              const name = file.filePath.slice(prefix ? `${prefix}${path.sep}`.length : 0);
+              let mode = file.stat.mode;
+              if (filesToChmodPlusX && _.includes(filesToChmodPlusX, name)
+                  && file.stat.mode % 2 === 0) {
+                mode += 1;
+              }
+              zip.append(file.data, {
+                name,
+                mode,
+                date: new Date(0), // necessary to get the same hash when zipping the same content
+              });
             });
-          });
 
-          zip.finalize();
-        }).catch(reject);
+            zip.finalize();
+          }).catch(reject);
       });
     });
   },


Test patch:
diff --git a/lib/plugins/package/lib/packageService.test.js b/lib/plugins/package/lib/packageService.test.js
index d5c9ac1caee..f03b5e8c300 100644
--- a/lib/plugins/package/lib/packageService.test.js
+++ b/lib/plugins/package/lib/packageService.test.js
@@ -12,7 +12,7 @@ const serverlessConfigFileUtils = require('../../../../lib/utils/getServerlessCo
 // Configure chai
 chai.use(require('chai-as-promised'));
 chai.use(require('sinon-chai'));
-const expect = require('chai').expect;
+const { expect } = require('chai');
 
 describe('#packageService()', () => {
   let serverless;
@@ -516,6 +516,7 @@ describe('#packageService()', () => {
       ]));
     });
   });
+
   describe('#packageLayer()', () => {
     const exclude = ['test-exclude'];
     const include = ['test-include'];
diff --git a/lib/plugins/package/lib/zipService.test.js b/lib/plugins/package/lib/zipService.test.js
index 04ea026306e..82823f69d4f 100644
--- a/lib/plugins/package/lib/zipService.test.js
+++ b/lib/plugins/package/lib/zipService.test.js
@@ -1036,6 +1036,30 @@ describe('zipService', () => {
         });
     });
 
+    it('should include files only once', () => {
+      params.zipFileName = getTestArtifactFileName('include-outside-working-dir');
+      serverless.config.servicePath = path.join(serverless.config.servicePath, 'lib');
+      params.exclude = [
+        './**',
+      ];
+      params.include = [
+        '.././bin/**',
+      ];
+
+      return expect(packagePlugin.zip(params)).to.eventually.be
+        .equal(path.join(serverless.config.servicePath, '.serverless', params.zipFileName))
+        .then(artifact => {
+          const data = fs.readFileSync(artifact);
+          return expect(zip.loadAsync(data)).to.be.fulfilled;
+        }).then(unzippedData => {
+          const unzippedFileData = unzippedData.files;
+          expect(Object.keys(unzippedFileData).sort()).to.deep.equal([
+            'bin/binary-444',
+            'bin/binary-777',
+          ]);
+        });
+    });
+
     it('should throw an error if no files are matched', () => {
       params.exclude = ['**/**'];
       params.include = [];
diff --git a/tests/utils/fs/index.js b/tests/utils/fs/index.js
index e7a93f55b93..e1d1b42b980 100644
--- a/tests/utils/fs/index.js
+++ b/tests/utils/fs/index.js
@@ -19,6 +19,12 @@ function getTmpFilePath(fileName) {
   return path.join(getTmpDirPath(), fileName);
 }
 
+function createTmpDir() {
+  const dirPath = getTmpDirPath();
+  fse.ensureDirSync(dirPath);
+  return dirPath;
+}
+
 function createTmpFile(name) {
   const filePath = getTmpFilePath(name);
   fse.ensureFileSync(filePath);
@@ -50,6 +56,7 @@ module.exports = {
   tmpDirCommonPath,
   getTmpDirPath,
   getTmpFilePath,
+  createTmpDir,
   createTmpFile,
   replaceTextInFile,
   readYamlFile,