Repository: serverless/serverless (JavaScript)
Date: 2019-04-10 13:47:42+00:00
Task type: Feature

Issue:
AWS lambda layers deployment does not keep old version
<!--
1. If you have a question and not a bug report please ask first at http://forum.serverless.com
2. Please check if an issue already exists. This bug may have already been documented
3. Check out and follow our Guidelines: https://github.com/serverless/serverless/blob/master/CONTRIBUTING.md
4. Fill out the whole template so we have a good overview on the issue
5. Do not remove any section of the template. If something is not applicable leave it empty but leave it in the Issue
6. Please follow the template, otherwise we'll have to ask you to update it
-->

# This is a Bug Report

## Description

* What went wrong?
Old layer got deleted
* What did you expect should have happened?
Old layer should remain by default
* What was the config you used?
* What stacktrace or error message from your provider did you see?

Similar or dependent issues:
* #12345

## Additional Data

* ***Serverless Framework Version you're using***:
1.35.1
* ***Operating System***:
MacOS
* ***Stack Trace***:
* ***Provider Error messages***:



Issue discussion:
Please correct me if im wrong. During a update action, CF will create a new `Lambda:LayerVersion` and replace(delete) the previous one regardless of `DeletePolicy`. So at this moment, old layer version cannot be keep due to issue of CF.
I didn't analyze it deeper, but it looks like SAM can handle that.
Anyways this should be resolved otherwise it is unusable. Layer is shared
with others, even from different accounts.
They would be surprised to see lambda failing just because we've released a
new version of the layer.
They should be able to continue using old version and update when they want
to

пт, 1 февр. 2019 г., 11:13 johnwongapi notifications@github.com:

> Please correct me if im wrong. During a update action, CF will create a
> new Lambda:LayerVersion and replace(delete) the previous one regardless
> of DeletePolicy. So at this moment, old layer version cannot be keep due
> to issue of CF.
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/serverless/serverless/issues/5765#issuecomment-459618342>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AAfTwjFYKTizsJKi2v53DU8iEECqpslkks5vI9r_gaJpZM4abl9W>
> .
>

thanks @kodart , i found out how SAM do this: [link](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-template.html#serverless-sam-template-layerversion)

> When a serverless layer version is transformed, AWS SAM also transforms the logical ID of the resource so that old layer versions aren't automatically deleted by AWS CloudFormation when the resource is updated.

It is possible to do it by serverless🎉🎉

Agreed that it is possible via that technique @johnwongapi, but I fail to see the real advantage to doing so.

One downside AFAICT is that it would make it difficult to use sls to publish layers meant for consumption in a different service since the name to export wouldn't be predicable.

The other, not so much downside, but lack of upside, is that if you are using the layer within the same service, you still have odd name issues potentially. But more importantly, you'll still be publishing a new version of the layer everytime you deploy, so I'm not sure what the benefit in that case is of old layers.
@dschep logical id of a resource and layer name are two different things. logical id is only used by CF to update the resources it manages. 
when we create layer version, we don't need to remove old resource, so logical id should be different.
> so I'm not sure what the benefit in that case is of old layers

@dschep if you have a lambda function that uses that old layer version, it will start failing.
> @dschep logical id of a resource and layer name are two different things. logical id is only used by CF to update the resources it manages.

true, but CF refs & exports are a useful way to use layers

> @dschep if you have a lambda function that uses that old layer version, it will start failing.

huh, I'll have to look into that. That's not what I remember being told by AWS when layers were first introduced.
> > @dschep if you have a lambda function that uses that old layer version, it will start failing.
> 
> huh, I'll have to look into that. That's not what I remember being told by AWS when layers were first introduced.

@dschep if the layer version is deleted, lambda is still runnable with it. But there will be a error when having code change to the lambda.
This should be resolved by having serverless always create layer version resources with a deletion policy of retain, and changing the name when generating a new layer.
I think this error is related on `sls deploy` with an `Output` resource:

```
  Serverless Error ---------------------------------------
 
  An error occurred: my-dev - Export MyLayerArn cannot be updated as it is in use by my-other-stack-dev.
```

Currently, the way layers are being destroyed on update prevents cross-stack imports entirely:

```
functions:
  my-function:
    ...
    layers:
      - 'Fn::ImportValue': MyLayerArn
```
+1 to getting this behaviour changed
I think this behavior needs to change, or at least give us the choice to retain old layer.

Solution patch:
diff --git a/docs/providers/aws/guide/layers.md b/docs/providers/aws/guide/layers.md
index d0920d6c364..fa895be1c71 100644
--- a/docs/providers/aws/guide/layers.md
+++ b/docs/providers/aws/guide/layers.md
@@ -36,6 +36,7 @@ layers:
     licenseInfo: GPLv3 # optional, a string specifying license information
     allowedAccounts: # optional, a list of AWS account IDs allowed to access this layer.
       - '*'
+    retain: false # optional, false by default. If true, layer versions are not deleted as new ones are created
 ```
 
 You can add up to 5 layers as you want within this property.
diff --git a/docs/providers/aws/guide/serverless.yml.md b/docs/providers/aws/guide/serverless.yml.md
index a2f3f49a704..c9e33677d64 100644
--- a/docs/providers/aws/guide/serverless.yml.md
+++ b/docs/providers/aws/guide/serverless.yml.md
@@ -262,6 +262,18 @@ functions:
           pool: MyUserPool
           trigger: PreSignUp
 
+layers:
+  hello: # A Lambda layer
+    path: layer-dir # required, path to layer contents on disk
+    name: ${self:provider.stage}-layerName # optional, Deployed Lambda layer name
+    description: Description of what the lambda layer does # optional, Description to publish to AWS
+    compatibleRuntimes: # optional, a list of runtimes this layer is compatible with
+      - python3.7
+    licenseInfo: GPLv3 # optional, a string specifying license information
+    allowedAccounts: # optional, a list of AWS account IDs allowed to access this layer.
+      - '*'
+    retain: false # optional, false by default. If true, layer versions are not deleted as new ones are created
+
 # The "Resources" your "Functions" use.  Raw AWS CloudFormation goes in here.
 resources:
   Resources:
diff --git a/lib/plugins/aws/package/compile/layers/index.js b/lib/plugins/aws/package/compile/layers/index.js
index d7de6f554fc..d79dd29cccb 100644
--- a/lib/plugins/aws/package/compile/layers/index.js
+++ b/lib/plugins/aws/package/compile/layers/index.js
@@ -1,5 +1,6 @@
 'use strict';
 
+const crypto = require('crypto');
 const BbPromise = require('bluebird');
 const _ = require('lodash');
 const path = require('path');
@@ -49,7 +50,12 @@ class AwsCompileLayers {
       newLayer.Properties.CompatibleRuntimes = layerObject.compatibleRuntimes;
     }
 
-    const layerLogicalId = this.provider.naming.getLambdaLayerLogicalId(layerName);
+    let layerLogicalId = this.provider.naming.getLambdaLayerLogicalId(layerName);
+    if (layerObject.retain) {
+      const sha = crypto.createHash('sha1').update(JSON.stringify(newLayer)).digest('hex');
+      layerLogicalId = `${layerLogicalId}${sha}`;
+      newLayer.DeletionPolicy = 'Retain';
+    }
     const newLayerObject = {
       [layerLogicalId]: newLayer,
     };


Test patch:
diff --git a/lib/plugins/aws/package/compile/layers/index.test.js b/lib/plugins/aws/package/compile/layers/index.test.js
index 6bf8dd41419..c9ec6612408 100644
--- a/lib/plugins/aws/package/compile/layers/index.test.js
+++ b/lib/plugins/aws/package/compile/layers/index.test.js
@@ -1,5 +1,6 @@
 'use strict';
 
+const crypto = require('crypto');
 const path = require('path');
 const chai = require('chai');
 const AwsProvider = require('../../../provider/awsProvider');
@@ -118,6 +119,48 @@ describe('AwsCompileLayers', () => {
       });
     });
 
+    it('should create a layer resource with a retention policy', () => {
+      const s3Folder = awsCompileLayers.serverless.service.package.artifactDirectoryName;
+      const s3FileName = awsCompileLayers.serverless.service.layers.test.package.artifact
+        .split(path.sep).pop();
+      awsCompileLayers.serverless.service.layers = {
+        test: {
+          path: 'layer',
+          retain: true,
+        },
+      };
+      const compiledLayer = {
+        Type: 'AWS::Lambda::LayerVersion',
+        Properties: {
+          Content: {
+            S3Bucket: { Ref: 'ServerlessDeploymentBucket' },
+            S3Key: `${s3Folder}/${s3FileName}`,
+          },
+          LayerName: 'test',
+        },
+      };
+      const sha = crypto.createHash('sha1').update(JSON.stringify(compiledLayer)).digest('hex');
+      compiledLayer.DeletionPolicy = 'Retain';
+      const compiledLayerOutput = {
+        Description: 'Current Lambda layer version',
+        Value: {
+          Ref: `TestLambdaLayer${sha}`,
+        },
+      };
+
+      return expect(awsCompileLayers.compileLayers()).to.be.fulfilled
+      .then(() => {
+        expect(
+          awsCompileLayers.serverless.service.provider.compiledCloudFormationTemplate
+          .Resources[`TestLambdaLayer${sha}`]
+        ).to.deep.equal(compiledLayer);
+        expect(
+          awsCompileLayers.serverless.service.provider.compiledCloudFormationTemplate
+            .Outputs.TestLambdaLayerQualifiedArn
+        ).to.deep.equal(compiledLayerOutput);
+      });
+    });
+
     it('should create a layer resource with permissions', () => {
       const s3Folder = awsCompileLayers.serverless.service.package.artifactDirectoryName;
       const s3FileName = awsCompileLayers.serverless.service.layers.test.package.artifact