Repository: serverless/serverless (JavaScript)
Date: 2020-02-03 13:52:52+00:00
Task type: Feature

Issue:
Make number of deployment artifacts to keep configurable.
# Feature Proposal

## Description

It would be nice feature if we could configure number of deployment artifacts to keep on S3 bucket. Currently this number is [hard-coded to 5](https://github.com/serverless/serverless/blob/master/lib/plugins/aws/deploy/lib/cleanupS3Bucket.js#L10).

We have CI deployment job which makes deployment and after that runs smoke tests. These smoke tests tend to be flaky (depend on 3rd party service), and this causes multiple `serverless deploy`'s to be executed until these tests pass. Problem is that if we have to repeat the deployment 5 times or more we lose our ability to rollback to previous good deployment because it was already cleaned up by [this script](https://github.com/serverless/serverless/blob/master/lib/plugins/aws/deploy/lib/cleanupS3Bucket.js).

This feature could be implemented without breaking BC. There could be new optional configuration file property (for example `deploymentArtifactsToKeep`) which would have default value of 5.



Issue discussion:
@eeroniemi thanks for proposal. It's definitely a good idea.

I think we can support tweaking it directly in `serverless.yml` as e.g.

```yaml
provider:
   preservedLastStackVersionsCount: 5 # default
```

PR that implements it, is definitely welcome!

Solution patch:
diff --git a/docs/providers/aws/guide/serverless.yml.md b/docs/providers/aws/guide/serverless.yml.md
index 322fdd3e215..f2b34d70bd5 100644
--- a/docs/providers/aws/guide/serverless.yml.md
+++ b/docs/providers/aws/guide/serverless.yml.md
@@ -41,6 +41,7 @@ provider:
   logRetentionInDays: 14 # Set the default RetentionInDays for a CloudWatch LogGroup
   deploymentBucket:
     name: com.serverless.${self:provider.region}.deploys # Deployment bucket name. Default is generated by the framework
+    maxPreviousDeploymentArtifacts: 10 # On every deployment the framework prunes the bucket to remove artifacts older than this limit. The default is 5
     blockPublicAccess: true # Prevents public access via ACLs or bucket policies. Default is false
     serverSideEncryption: AES256 # server-side encryption method
     sseKMSKeyId: arn:aws:kms:us-east-1:xxxxxxxxxxxx:key/aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa # when using server-side encryption
diff --git a/lib/plugins/aws/deploy/lib/cleanupS3Bucket.js b/lib/plugins/aws/deploy/lib/cleanupS3Bucket.js
index 8de368159cd..0a8cce58abf 100644
--- a/lib/plugins/aws/deploy/lib/cleanupS3Bucket.js
+++ b/lib/plugins/aws/deploy/lib/cleanupS3Bucket.js
@@ -7,7 +7,12 @@ const getS3ObjectsFromStacks = require('../../utils/getS3ObjectsFromStacks');
 
 module.exports = {
   getObjectsToRemove() {
-    const stacksToKeepCount = 5;
+    const stacksToKeepCount = _.get(
+      this.serverless,
+      'service.provider.deploymentBucketObject.maxPreviousDeploymentArtifacts',
+      5
+    );
+
     const service = this.serverless.service.service;
     const stage = this.provider.getStage();
     const prefix = this.provider.getDeploymentPrefix();


Test patch:
diff --git a/lib/plugins/aws/deploy/lib/cleanupS3Bucket.test.js b/lib/plugins/aws/deploy/lib/cleanupS3Bucket.test.js
index abd44b8ad16..e66fe343e37 100644
--- a/lib/plugins/aws/deploy/lib/cleanupS3Bucket.test.js
+++ b/lib/plugins/aws/deploy/lib/cleanupS3Bucket.test.js
@@ -133,7 +133,7 @@ describe('cleanupS3Bucket', () => {
       });
     });
 
-    it('should resolve if there are exactly 4 directories available', () => {
+    it('should return an empty array if there are exactly 4 directories available', () => {
       const serviceObjects = {
         Contents: [
           { Key: `${s3Key}151224711231-2016-08-18T15:42:00/artifact.zip` },
@@ -159,6 +159,57 @@ describe('cleanupS3Bucket', () => {
         awsDeploy.provider.request.restore();
       });
     });
+
+    describe('custom maxPreviousDeploymentArtifacts', () => {
+      afterEach(() => {
+        // restore to not conflict with other tests
+        delete serverless.service.provider.deploymentBucketObject;
+      });
+
+      it('should allow configuring the number of artifacts to preserve', () => {
+        // configure the provider to allow only a single artifact
+        serverless.service.provider.deploymentBucketObject = {
+          maxPreviousDeploymentArtifacts: 1,
+        };
+
+        const serviceObjects = {
+          Contents: [
+            { Key: `${s3Key}/141321321541-2016-08-18T11:23:02/artifact.zip` },
+            { Key: `${s3Key}/141321321541-2016-08-18T11:23:02/cloudformation.json` },
+            { Key: `${s3Key}/151224711231-2016-08-18T15:42:00/artifact.zip` },
+            { Key: `${s3Key}/151224711231-2016-08-18T15:42:00/cloudformation.json` },
+            { Key: `${s3Key}/141264711231-2016-08-18T15:43:00/artifact.zip` },
+            { Key: `${s3Key}/141264711231-2016-08-18T15:43:00/cloudformation.json` },
+          ],
+        };
+
+        const listObjectsStub = sinon.stub(awsDeploy.provider, 'request').resolves(serviceObjects);
+
+        return awsDeploy.getObjectsToRemove().then(objectsToRemove => {
+          expect(objectsToRemove).to.deep.include.members([
+            { Key: `${s3Key}/141321321541-2016-08-18T11:23:02/artifact.zip` },
+            { Key: `${s3Key}/141321321541-2016-08-18T11:23:02/cloudformation.json` },
+            { Key: `${s3Key}/151224711231-2016-08-18T15:42:00/artifact.zip` },
+            { Key: `${s3Key}/151224711231-2016-08-18T15:42:00/cloudformation.json` },
+          ]);
+
+          expect(objectsToRemove).to.not.deep.include({
+            Key: `${s3Key}/141264711231-2016-08-18T15:43:00/artifact.zip`,
+          });
+
+          expect(objectsToRemove).to.not.deep.include({
+            Key: `${s3Key}/141264711231-2016-08-18T15:43:00/cloudformation.json`,
+          });
+
+          expect(listObjectsStub.calledOnce).to.be.equal(true);
+          expect(listObjectsStub).to.have.been.calledWithExactly('S3', 'listObjectsV2', {
+            Bucket: awsDeploy.bucketName,
+            Prefix: `${s3Key}`,
+          });
+          awsDeploy.provider.request.restore();
+        });
+      });
+    });
   });
 
   describe('#removeObjects()', () => {