Repository: serverless/serverless (JavaScript)
Date: 2019-08-09 02:01:52+00:00
Task type: Bug Fix

Issue:
Ignore check for unchanged files after failed deployment
After a failed deployment due to remote (CloudFormation) error I have to use `sls deploy --force` to trigger a new deployment with the same configuration because serverless skips the deployment with the `Service files not changed. Skipping deployment...` notice per default.

### Expected behaviour

If a deployment failed due to a remote (CloudFormation) error, a new `sls deploy` should trigger a deployment even if the service files have not changed. The check for unchanged configuration should only matter after a successful deployment.



Issue discussion:
@sbstjn 
Good point. I agree with you.
This is kind of silly... I need to change my project name to another one if it fails in order to bypass this wrong checking.
Have you tried to use `sls deploy --force` @tom10271 ?
Funny thing: The --force flag ignores forced webpack excludes and thus fails due to missing packages.
I’ve encountered this issue numerous times now when a previous deploy failed because of a misconfiguration. It’s really confusing when deploys seem to be successful, but they’re really quick and then nestled in between a bunch of other output text I see:

```bash
Serverless: Service files not changed. Skipping deployment...
```


Just ran into this, myself. The `--force` flag works fine, but this should probably still be addressed.

When a Cloudformation deploy fails, it rolls back any changes that were attempted during deploy. Because of this, on a failed deploy the stored serverless "state" in S3 has diverged from the actual state of the AWS service/Cloudformation stack. So not only does it make re-deploying annoyingly require the `--force` flag, but it means there's drift between cached state and actual state. It's not too hard to imagine this causing issues down the road if someone were to build other features that utilized the cached "state" in S3.

It seems like serverless should either clean up the artifacts on deployment failure or add another artifact to the S3 location that reports on deployment result (success/fail) so that the framework can make more intelligent decisions around the actual state of the service. 
+1 Bump on this issue. Serverless should be able to clean up failed deployments. Using `--force` doesn't seem like a very stable solution.
the solution without --force would be a great addition

I would also say given a lot of releases looking like a major upgrade to me, is there any plan to bump up SLS to 2.x? @pmuens 
+1 bump on this
Ran into this while configuring circleci for Serverless deployments, one can always expect lots of build fails going from zero to continuous.

+1 please

**EDIT**

I've set our CI/CD system to always run with `--force`.  Just too much room for confusion otherwise.  Can't be wasting hours tracking down an obscure message in the circle logs.  Thankfully our Webpack config isn't that complicated yet.

Question, what is the use case for running w/o `--force`?  To ask it another way, why would we ever run `sls deploy` and not want it to actually deploy?
> Question, what is the use case for running w/o --force? To ask it another way, why would we ever run sls deploy and not want it to actually deploy?

deploying to lambda@edge counts as a cloudfront distribution change which takes ages.
if we can avoid doing that unnecessarily, e.g. just deploy the lambdas that have actually changed that'd be a big help.

Solution patch:
diff --git a/lib/plugins/aws/deploy/lib/checkForChanges.js b/lib/plugins/aws/deploy/lib/checkForChanges.js
index ad71b3c4b15..cf03a3f885e 100644
--- a/lib/plugins/aws/deploy/lib/checkForChanges.js
+++ b/lib/plugins/aws/deploy/lib/checkForChanges.js
@@ -18,8 +18,15 @@ module.exports = {
 
     return BbPromise.bind(this)
       .then(this.getMostRecentObjects)
-      .then(this.getObjectMetadata)
-      .then(this.checkIfDeploymentIsNecessary)
+      .then(objs => {
+        return BbPromise.all([
+          this.getObjectMetadata(objs),
+          this.getFunctionsEarliestLastModifiedDate(),
+        ]);
+      })
+      .then(([objMetadata, lastModifiedDate]) =>
+        this.checkIfDeploymentIsNecessary(objMetadata, lastModifiedDate)
+      )
       .then(() => {
         if (this.serverless.service.provider.shouldNotDeploy) {
           return BbPromise.resolve();
@@ -78,6 +85,23 @@ module.exports = {
       });
   },
 
+  // Gives the least recent last modify date accross all the functions in the service.
+  getFunctionsEarliestLastModifiedDate() {
+    const getFunctionResults = this.serverless.service.getAllFunctions().map(funName => {
+      const functionObj = this.serverless.service.getFunction(funName);
+      return this.provider
+        .request('Lambda', 'getFunction', {
+          FunctionName: functionObj.name,
+        })
+        .then(res => new Date(res.Configuration.LastModified))
+        .catch(() => new Date(0)); // Function is missing, needs to be deployed
+    });
+
+    return BbPromise.all(getFunctionResults)
+      .then(_.min)
+      .then(x => x || null);
+  },
+
   getObjectMetadata(objects) {
     if (objects && objects.length) {
       const headObjectObjects = objects.map(obj =>
@@ -93,7 +117,7 @@ module.exports = {
     return BbPromise.resolve([]);
   },
 
-  checkIfDeploymentIsNecessary(objects) {
+  checkIfDeploymentIsNecessary(objects, funcLastModifiedDate) {
     if (objects && objects.length) {
       const remoteHashes = objects.map(object => object.Metadata.filesha256 || '');
 
@@ -125,7 +149,13 @@ module.exports = {
         const localHashes = zipFileHashes;
         localHashes.push(localCfHash);
 
-        if (_.isEqual(remoteHashes.sort(), localHashes.sort())) {
+        // If any objects were changed after the last time the function was updated
+        // there could have been a failed deploy.
+        const changedAfterDeploy = _.some(objects, object => {
+          return object.LastModified && object.LastModified > funcLastModifiedDate;
+        });
+
+        if (!changedAfterDeploy && _.isEqual(remoteHashes.sort(), localHashes.sort())) {
           this.serverless.service.provider.shouldNotDeploy = true;
 
           const message = ['Service files not changed. Skipping deployment...'].join('');


Test patch:
diff --git a/lib/plugins/aws/deploy/lib/checkForChanges.test.js b/lib/plugins/aws/deploy/lib/checkForChanges.test.js
index 915f072497e..a3bbf0bc61e 100644
--- a/lib/plugins/aws/deploy/lib/checkForChanges.test.js
+++ b/lib/plugins/aws/deploy/lib/checkForChanges.test.js
@@ -284,6 +284,36 @@ describe('checkForChanges', () => {
       });
     });
 
+    it('should resolve if objects are given, but no function last modified date', () => {
+      globbySyncStub.returns(['my-service.zip']);
+      cryptoStub
+        .createHash()
+        .update()
+        .digest.onCall(0)
+        .returns('local-hash-cf-template');
+
+      const input = [{ Metadata: { filesha256: 'remote-hash-cf-template' } }];
+
+      return expect(awsDeploy.checkIfDeploymentIsNecessary(input)).to.be.fulfilled.then(() => {
+        expect(normalizeCloudFormationTemplateStub).to.have.been.calledOnce;
+        expect(globbySyncStub).to.have.been.calledOnce;
+        expect(readFileStub).to.have.been.calledOnce;
+        expect(awsDeploy.serverless.cli.log).to.not.have.been.called;
+        expect(normalizeCloudFormationTemplateStub).to.have.been.calledWithExactly(
+          awsDeploy.serverless.service.provider.compiledCloudFormationTemplate
+        );
+        expect(globbySyncStub).to.have.been.calledWithExactly(['**.zip'], {
+          cwd: path.join(awsDeploy.serverless.config.servicePath, '.serverless'),
+          dot: true,
+          silent: true,
+        });
+        expect(readFileStub).to.have.been.calledWith(
+          path.join('my-service/.serverless/my-service.zip')
+        );
+        expect(awsDeploy.serverless.service.provider.shouldNotDeploy).to.equal(undefined);
+      });
+    });
+
     it('should not set a flag if there are more remote hashes', () => {
       globbySyncStub.returns(['my-service.zip']);
       cryptoStub
@@ -408,6 +438,48 @@ describe('checkForChanges', () => {
       });
     });
 
+    it('should not set a flag if the hashes are equal, but the objects were modified after their functions', () => {
+      globbySyncStub.returns(['my-service.zip']);
+      cryptoStub
+        .createHash()
+        .update()
+        .digest.onCall(0)
+        .returns('hash-cf-template');
+      cryptoStub
+        .createHash()
+        .update()
+        .digest.onCall(1)
+        .returns('hash-zip-file-1');
+
+      const now = new Date();
+      const inThePast = new Date(new Date().getTime() - 100000);
+      const inTheFuture = new Date(new Date().getTime() + 100000);
+
+      const input = [
+        { Metadata: { filesha256: 'hash-cf-template' }, LastModified: inThePast },
+        { Metadata: { filesha256: 'hash-zip-file-1' }, LastModified: inTheFuture },
+      ];
+
+      return expect(awsDeploy.checkIfDeploymentIsNecessary(input, now)).to.be.fulfilled.then(() => {
+        expect(normalizeCloudFormationTemplateStub).to.have.been.calledOnce;
+        expect(globbySyncStub).to.have.been.calledOnce;
+        expect(readFileStub).to.have.been.calledOnce;
+        expect(awsDeploy.serverless.cli.log).to.not.have.been.called;
+        expect(normalizeCloudFormationTemplateStub).to.have.been.calledWithExactly(
+          awsDeploy.serverless.service.provider.compiledCloudFormationTemplate
+        );
+        expect(globbySyncStub).to.have.been.calledWithExactly(['**.zip'], {
+          cwd: path.join(awsDeploy.serverless.config.servicePath, '.serverless'),
+          dot: true,
+          silent: true,
+        });
+        expect(readFileStub).to.have.been.calledWith(
+          path.join('my-service/.serverless/my-service.zip')
+        );
+        expect(awsDeploy.serverless.service.provider.shouldNotDeploy).to.equal(undefined);
+      });
+    });
+
     it('should set a flag if the remote and local hashes are equal', () => {
       globbySyncStub.returns(['my-service.zip']);
       cryptoStub
@@ -446,6 +518,49 @@ describe('checkForChanges', () => {
       });
     });
 
+    it('should set a flag if the remote and local hashes are equal, and the edit times are ordered', () => {
+      globbySyncStub.returns(['my-service.zip']);
+      cryptoStub
+        .createHash()
+        .update()
+        .digest.onCall(0)
+        .returns('hash-cf-template');
+      cryptoStub
+        .createHash()
+        .update()
+        .digest.onCall(1)
+        .returns('hash-zip-file-1');
+
+      const longAgo = new Date(new Date().getTime() - 100000);
+      const longerAgo = new Date(new Date().getTime() - 200000);
+
+      const input = [
+        { Metadata: { filesha256: 'hash-cf-template' }, LastModified: longerAgo },
+        { Metadata: { filesha256: 'hash-zip-file-1' }, LastModified: longerAgo },
+      ];
+
+      return expect(awsDeploy.checkIfDeploymentIsNecessary(input, longAgo)).to.be.fulfilled.then(
+        () => {
+          expect(normalizeCloudFormationTemplateStub).to.have.been.calledOnce;
+          expect(globbySyncStub).to.have.been.calledOnce;
+          expect(readFileStub).to.have.been.calledOnce;
+          expect(awsDeploy.serverless.cli.log).to.have.been.calledOnce;
+          expect(normalizeCloudFormationTemplateStub).to.have.been.calledWithExactly(
+            awsDeploy.serverless.service.provider.compiledCloudFormationTemplate
+          );
+          expect(globbySyncStub).to.have.been.calledWithExactly(['**.zip'], {
+            cwd: path.join(awsDeploy.serverless.config.servicePath, '.serverless'),
+            dot: true,
+            silent: true,
+          });
+          expect(readFileStub).to.have.been.calledWith(
+            path.join('my-service/.serverless/my-service.zip')
+          );
+          expect(awsDeploy.serverless.service.provider.shouldNotDeploy).to.equal(true);
+        }
+      );
+    });
+
     it('should set a flag if the remote and local hashes are duplicated and equal', () => {
       globbySyncStub.returns(['func1.zip', 'func2.zip']);
       cryptoStub
@@ -741,5 +856,100 @@ describe('checkForChanges', () => {
           .then(() => expect(deleteSubscriptionFilterStub).to.have.been.called);
       });
     });
+
+    describe('#getFunctionsLatestLastModifiedDate', () => {
+      let getFunctionStub;
+
+      beforeEach(() => {
+        getFunctionStub = sandbox.stub(awsDeploy.provider, 'request').resolves();
+      });
+
+      afterEach(() => {
+        awsDeploy.provider.request.restore();
+      });
+
+      it('should return null when there are no functions', () => {
+        awsDeploy.serverless.service.functions = {};
+        awsDeploy.serverless.service.setFunctionNames();
+
+        return expect(awsDeploy.getFunctionsEarliestLastModifiedDate()).to.have.been.fulfilled.then(
+          ans => {
+            expect(getFunctionStub).to.have.not.been.called;
+            expect(ans).to.be.null;
+          }
+        );
+      });
+
+      it('should treat rejections as epoch', () => {
+        awsDeploy.provider.request.restore();
+
+        getFunctionStub = sandbox.stub(awsDeploy.provider, 'request');
+
+        const now = new Date();
+        getFunctionStub.onCall(0).returns(BbPromise.reject());
+        getFunctionStub
+          .onCall(1)
+          .returns(BbPromise.resolve({ Configuration: { LastModified: now } }));
+
+        awsDeploy.serverless.service.functions = {
+          first: {
+            events: [{ someevent: 'abc' }],
+          },
+          second: {
+            events: [{ anothaone: '1' }],
+          },
+        };
+
+        awsDeploy.serverless.service.setFunctionNames();
+
+        return expect(awsDeploy.getFunctionsEarliestLastModifiedDate()).to.have.been.fulfilled.then(
+          ans => {
+            expect(ans.valueOf()).to.equal(new Date(0).valueOf());
+            expect(getFunctionStub).to.have.been.calledTwice;
+          }
+        );
+      });
+
+      it('should return the earliest last modified date', () => {
+        awsDeploy.provider.request.restore();
+
+        getFunctionStub = sandbox.stub(awsDeploy.provider, 'request');
+
+        const now = new Date();
+        const longAgo = new Date(new Date().getTime() - 100000);
+        const longerAgo = new Date(new Date().getTime() - 100001);
+
+        getFunctionStub
+          .onCall(0)
+          .returns(BbPromise.resolve({ Configuration: { LastModified: longAgo } }));
+        getFunctionStub
+          .onCall(1)
+          .returns(BbPromise.resolve({ Configuration: { LastModified: longerAgo } }));
+        getFunctionStub
+          .onCall(2)
+          .returns(BbPromise.resolve({ Configuration: { LastModified: now } }));
+
+        awsDeploy.serverless.service.functions = {
+          first: {
+            events: [{ someevent: 'abc' }],
+          },
+          second: {
+            events: [{ anothaone: '1' }],
+          },
+          third: {
+            events: [{ thebest: 'around' }],
+          },
+        };
+
+        awsDeploy.serverless.service.setFunctionNames();
+
+        return expect(awsDeploy.getFunctionsEarliestLastModifiedDate()).to.have.been.fulfilled.then(
+          ans => {
+            expect(ans.valueOf()).to.equal(longerAgo.valueOf());
+            expect(getFunctionStub).to.have.been.calledThrice;
+          }
+        );
+      });
+    });
   });
 });