Repository: serverless/serverless (JavaScript)
Date: 2019-08-14 15:12:20+00:00
Task type: Bug Fix

Issue:
Unnecessary deployments with retained lambda layer
<!--
1. If you have a question and not a bug report please ask first at http://forum.serverless.com
2. Please check if an issue already exists. This bug may have already been documented
3. Check out and follow our Guidelines: https://github.com/serverless/serverless/blob/master/CONTRIBUTING.md
4. Fill out the whole template so we have a good overview on the issue
5. Do not remove any section of the template. If something is not applicable leave it empty but leave it in the Issue
6. Please follow the template, otherwise we'll have to ask you to update it
-->

# This is a Bug Report

## Description

- What went wrong?
If you specify "retain" in the LambdaLayer, the deployment is performed without modifying the source.
- What did you expect should have happened?
I expect `sls deploy` to skip deployment, when without modifying the source.
- What was the config you used?
lambda layer with retain
- What stacktrace or error message from your provider did you see?

Similar or dependent issues:

- #5765

## Additional Data

- **_Serverless Framework Version you're using_**: 1.49.0 (Enterprise Plugin: 1.3.8, Platform SDK: 2.1.0)
- **_Operating System_**: macOS Mojave 10.14.4
- **_Stack Trace_**: -
- **_Provider Error messages_**: -

I think that the suffix sha for layerLogicalId should be computed with the exception of S3Key.
https://github.com/serverless/serverless/blob/1f1e5c05efb8f061a8b12393e327a9c764d5bbaa/lib/plugins/aws/package/compile/layers/index.js#L54-L61


Issue discussion:
@mokamoto12 thanks for report. Still I think deployment in such case is expected.

While nothing resource-wise was changed, the CloudFormation instruction have been changed and that has implications on its own.

Imagine the scenario, when after deployment we decide to remove the stack, if prior deployment was skipped, the LambdaLayer will be deleted as _DeletionPolicy_ was not recorded on AWS side.

@medikoo Please allow me to check that my understanding is correct or not.
Is the case you are talking about the following flow?

1. Deploy without layer retention
2. Configure to retain the layer and deploy it

I'm sorry if the understanding is wrong. I think it should be deployed in this case.
However, I don't think you need to deploy in the following cases.

1. Deploy with layer retention
2. Deploy as the layer is retained without modifying the source code

I'm sorry if something went wrong.
In following:

> 1. Deploy with layer retention
> 2. Deploy as the layer is retained without modifying the source code

Do you suggest that no configuration changes are made (at 2) after deployment (1) and still further depoyment is fully made?
If not, please clarify _"Deploy as the layer is retained without modifying the source code"_ as it's not clear to me.
@medikoo Sorry, I rewrite it as follows.

I don't think you need to deploy in the following cases.
1. `sls deploy` with layer retention
2. `sls deploy` with layer retention, without modifying the source code

I don't think the deployment should be done (at 2).
Am I making sense?
> I don't think the deployment should be done (at 2).

Ok, so you mean that doing two consecutive deploys (with no changes to configuration or source code between them), invokes two full stack deployments, while you expect the second one to be skipped right?

So we can also put it as:

_When layer retention is set, then deployment is never skipped_

Right?

@medikoo Thank you!

> Ok, so you mean that doing two consecutive deploys (with no changes to configuration or source code between them), invokes two full stack deployments, while you expect the second one to be skipped right?

Yes! I expect the second one to be skipped!
But It doesn't make sense to me about the following.

> When layer retention is set, then deployment is never skipped

I think this should also be skipped.
Is this wrong?
> But It doesn't make sense to me about the following.

Here I just explained the problem in other words, to ensure we have same understanding.

In all cases if no changes are made the deployment should be skipped, that's expected behavior

Solution patch:
diff --git a/lib/plugins/aws/package/compile/layers/index.js b/lib/plugins/aws/package/compile/layers/index.js
index b75e5ec88e0..f19b8279a62 100644
--- a/lib/plugins/aws/package/compile/layers/index.js
+++ b/lib/plugins/aws/package/compile/layers/index.js
@@ -54,7 +54,7 @@ class AwsCompileLayers {
     if (layerObject.retain) {
       const sha = crypto
         .createHash('sha1')
-        .update(JSON.stringify(newLayer))
+        .update(JSON.stringify(_.omit(newLayer, ['Properties.Content.S3Key'])))
         .digest('hex');
       layerLogicalId = `${layerLogicalId}${sha}`;
       newLayer.DeletionPolicy = 'Retain';


Test patch:
diff --git a/lib/plugins/aws/package/compile/layers/index.test.js b/lib/plugins/aws/package/compile/layers/index.test.js
index dfadb7011e9..9d9e9e03211 100644
--- a/lib/plugins/aws/package/compile/layers/index.test.js
+++ b/lib/plugins/aws/package/compile/layers/index.test.js
@@ -3,6 +3,7 @@
 const crypto = require('crypto');
 const path = require('path');
 const chai = require('chai');
+const _ = require('lodash');
 const AwsProvider = require('../../../provider/awsProvider');
 const AwsCompileLayers = require('./index');
 const Serverless = require('../../../../../Serverless');
@@ -148,7 +149,7 @@ describe('AwsCompileLayers', () => {
       };
       const sha = crypto
         .createHash('sha1')
-        .update(JSON.stringify(compiledLayer))
+        .update(JSON.stringify(_.omit(compiledLayer, ['Properties.Content.S3Key'])))
         .digest('hex');
       compiledLayer.DeletionPolicy = 'Retain';
       const compiledLayerOutput = {
@@ -171,6 +172,76 @@ describe('AwsCompileLayers', () => {
       });
     });
 
+    it('should create a layer resource with a retention policy, has the same logical id when not modify', () => {
+      awsCompileLayers.serverless.service.layers = {
+        test: {
+          path: 'layer',
+          retain: true,
+        },
+      };
+      const secondAwsCompileLayers = _.cloneDeep(awsCompileLayers);
+      secondAwsCompileLayers.serverless.service.package.artifactDirectoryName = 'somedir2';
+
+      return expect(
+        Promise.all([awsCompileLayers.compileLayers(), secondAwsCompileLayers.compileLayers()])
+      ).to.be.fulfilled.then(() => {
+        expect(
+          _.findKey(
+            awsCompileLayers.serverless.service.provider.compiledCloudFormationTemplate.Resources,
+            r => r.Type === 'AWS::Lambda::LayerVersion'
+          )
+        ).to.deep.equal(
+          _.findKey(
+            secondAwsCompileLayers.serverless.service.provider.compiledCloudFormationTemplate
+              .Resources,
+            r => r.Type === 'AWS::Lambda::LayerVersion'
+          )
+        );
+        expect(
+          awsCompileLayers.serverless.service.provider.compiledCloudFormationTemplate.Outputs
+            .TestLambdaLayerQualifiedArn
+        ).to.deep.equal(
+          secondAwsCompileLayers.serverless.service.provider.compiledCloudFormationTemplate.Outputs
+            .TestLambdaLayerQualifiedArn
+        );
+      });
+    });
+
+    it('should create a layer resource with a retention policy, has different logical id when modify', () => {
+      awsCompileLayers.serverless.service.layers = {
+        test: {
+          path: 'layer',
+          retain: true,
+        },
+      };
+      const secondAwsCompileLayers = _.cloneDeep(awsCompileLayers);
+      secondAwsCompileLayers.serverless.service.layers.test.description = 'modified description';
+
+      return expect(
+        Promise.all([awsCompileLayers.compileLayers(), secondAwsCompileLayers.compileLayers()])
+      ).to.be.fulfilled.then(() => {
+        expect(
+          _.findKey(
+            awsCompileLayers.serverless.service.provider.compiledCloudFormationTemplate.Resources,
+            r => r.Type === 'AWS::Lambda::LayerVersion'
+          )
+        ).to.not.deep.equal(
+          _.findKey(
+            secondAwsCompileLayers.serverless.service.provider.compiledCloudFormationTemplate
+              .Resources,
+            r => r.Type === 'AWS::Lambda::LayerVersion'
+          )
+        );
+        expect(
+          awsCompileLayers.serverless.service.provider.compiledCloudFormationTemplate.Outputs
+            .TestLambdaLayerQualifiedArn
+        ).to.not.deep.equal(
+          secondAwsCompileLayers.serverless.service.provider.compiledCloudFormationTemplate.Outputs
+            .TestLambdaLayerQualifiedArn
+        );
+      });
+    });
+
     it('should create a layer resource with permissions', () => {
       const s3Folder = awsCompileLayers.serverless.service.package.artifactDirectoryName;
       const s3FileName = awsCompileLayers.serverless.service.layers.test.package.artifact