Repository: serverless/serverless (JavaScript)
Date: 2020-01-02 16:10:28+00:00
Task type: Bug Fix

Issue:
serverless deploy fails with ROLLBACK_COMPLETE error
<!--
1. If you have a question and not a bug report please ask first at http://forum.serverless.com
2. Please check if an issue already exists. This bug may have already been documented
3. Check out and follow our Guidelines: https://github.com/serverless/serverless/blob/master/CONTRIBUTING.md
4. Fill out the whole template so we have a good overview on the issue
5. Do not remove any section of the template. If something is not applicable leave it empty but leave it in the Issue
6. Please follow the template, otherwise we'll have to ask you to update it
-->

# This is a Bug Report

## Description

- What went wrong?

I got the following error message upon trying to re-deploy my function after a previous error:

```
Serverless Error ---------------------------------------

  Stack:arn:aws:cloudformation:us-east-2:018790755515:stack/gmail-to-expensify-forwarding-service-dev/0ac0c780-cb27-11e9-9a01-0ab95d3a5528 is in ROLLBACK_COMPLETE state and can not be updated.
```

- What did you expect should have happened?

Serverless should have attempted to deploy the function.

- What was the config you used?

```
---
service:
  name: gmail-to-expensify-forwarding-service

frameworkVersion: '>=1.0.0'

provider:
  name: aws
  runtime: ruby2.5
  region: ${opt:region, 'us-east-2'}
  memorySize: 128
  deploymentBucket:
    name: ${env:SERVERLESS_BUCKET_NAME}
  deploymentPrefix: serverless

package:
  include:
    - lib/**
    - bin/**
    - .env

functions:
  forwarderBegin:
    handler: bin/forwarder.begin
    description: Begins scanning for and forwarding receipts.
    timeout: 60
    events:
      - schedule:
          name: Gmail Receipt check
          description: Checks for receipts
          rate: rate(1 minute)
          enabled: true
```
- What stacktrace or error message from your provider did you see?

```
Serverless: Packaging service...
Serverless: Excluding development dependencies...
Serverless: Uploading CloudFormation file to S3...
Serverless: Uploading artifacts...
Serverless: Uploading service gmail-to-expensify-forwarding-service.zip file to S3 (13.76 KB)...
Serverless: Validating template...
Serverless: Updating Stack...

  Serverless Error ---------------------------------------

  Stack:arn:aws:cloudformation:us-east-2:018790755515:stack/gmail-to-expensify-forwarding-service-dev/0ac0c780-cb27-11e9-9a01-0ab95d3a5528 is in ROLLBACK_COMPLETE state and can not be updated.

  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Issues:        forum.serverless.com

  Your Environment Information ---------------------------
     Operating System:          linux
     Node Version:              8.16.1
     Framework Version:         1.50.0
     Plugin Version:            1.3.8
     SDK Version:               2.1.0

```

Similar or dependent issues:

- #12345

## Additional Data

Using `awscli` to manually delete the CloudFormation stack seems to resolve the issue.

- **_Serverless Framework Version you're using_**:
- **_Operating System_**:
- **_Stack Trace_**:
- **_Provider Error messages_**:

serverless deploy fails with ROLLBACK_COMPLETE error
<!--
1. If you have a question and not a bug report please ask first at http://forum.serverless.com
2. Please check if an issue already exists. This bug may have already been documented
3. Check out and follow our Guidelines: https://github.com/serverless/serverless/blob/master/CONTRIBUTING.md
4. Fill out the whole template so we have a good overview on the issue
5. Do not remove any section of the template. If something is not applicable leave it empty but leave it in the Issue
6. Please follow the template, otherwise we'll have to ask you to update it
-->

# This is a Bug Report

## Description

- What went wrong?

I got the following error message upon trying to re-deploy my function after a previous error:

```
Serverless Error ---------------------------------------

  Stack:arn:aws:cloudformation:us-east-2:018790755515:stack/gmail-to-expensify-forwarding-service-dev/0ac0c780-cb27-11e9-9a01-0ab95d3a5528 is in ROLLBACK_COMPLETE state and can not be updated.
```

- What did you expect should have happened?

Serverless should have attempted to deploy the function.

- What was the config you used?

```
---
service:
  name: gmail-to-expensify-forwarding-service

frameworkVersion: '>=1.0.0'

provider:
  name: aws
  runtime: ruby2.5
  region: ${opt:region, 'us-east-2'}
  memorySize: 128
  deploymentBucket:
    name: ${env:SERVERLESS_BUCKET_NAME}
  deploymentPrefix: serverless

package:
  include:
    - lib/**
    - bin/**
    - .env

functions:
  forwarderBegin:
    handler: bin/forwarder.begin
    description: Begins scanning for and forwarding receipts.
    timeout: 60
    events:
      - schedule:
          name: Gmail Receipt check
          description: Checks for receipts
          rate: rate(1 minute)
          enabled: true
```
- What stacktrace or error message from your provider did you see?

```
Serverless: Packaging service...
Serverless: Excluding development dependencies...
Serverless: Uploading CloudFormation file to S3...
Serverless: Uploading artifacts...
Serverless: Uploading service gmail-to-expensify-forwarding-service.zip file to S3 (13.76 KB)...
Serverless: Validating template...
Serverless: Updating Stack...

  Serverless Error ---------------------------------------

  Stack:arn:aws:cloudformation:us-east-2:018790755515:stack/gmail-to-expensify-forwarding-service-dev/0ac0c780-cb27-11e9-9a01-0ab95d3a5528 is in ROLLBACK_COMPLETE state and can not be updated.

  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Issues:        forum.serverless.com

  Your Environment Information ---------------------------
     Operating System:          linux
     Node Version:              8.16.1
     Framework Version:         1.50.0
     Plugin Version:            1.3.8
     SDK Version:               2.1.0

```

Similar or dependent issues:

- #12345

## Additional Data

Using `awscli` to manually delete the CloudFormation stack seems to resolve the issue.

- **_Serverless Framework Version you're using_**:
- **_Operating System_**:
- **_Stack Trace_**:
- **_Provider Error messages_**:



Issue discussion:
Encountered same issue while deploying the serverless on AWS.
Internally serverless create an AWS cloud formation stack by the name of your serverless name.

In my case, it was in "ROLLBACK_COMPLETE" state, so I manually deleted the stack from AWS console & redeploy the serverless again and worked perfectly.

I hope it helps you.
Should serverless have done the stack removal and re-creation for users automatically? Or at least provide this as an option?

Edit:
Interesting to see that there's #5631 that's supposed to fix this issue. Then why are we still seeing this issue?
yeah I think serverless should have a way of dealing with this. For example, 'sls remove' should just take care of it automatically
Same behavior, all these days I was resorting to deleting the stack manually and rerunning my pipeline. But, it would be very cool to stay hands-off from AWS console in ROLLBACK_COMPLETE scenarios.

+1 to have this fixed.
@cfbao & @ramgrandhi, do you happen to use a dedicated deployment bucket in your serverless.yml file? I ran into the same problem, and I was puzzled by the fact that [this commit](https://github.com/serverless/serverless/pull/5631/commits/25164d884cf86f3d62350d473011594575066241) indeed looked like it was supposed to fix it.

It turns out that serverless has a sort of "lazy" stack creation, and the code is in a different file (updateStack.js, go figure). That one gets triggered when you deploy using an existing deployment bucket instead of the default setting of letting Serverless create the bucket for you. And in that case, the stack is still created with the ROLLBACK action on failure.

The fix is just the same as #5631, I'll submit a pull request shortly.
> @cfbao & @ramgrandhi, do you happen to use a dedicated deployment bucket in your serverless.yml file?

Yes, I do! Sounds like you've figured it out. Thanks in advance.
Encountered same issue while deploying the serverless on AWS.
Internally serverless create an AWS cloud formation stack by the name of your serverless name.

In my case, it was in "ROLLBACK_COMPLETE" state, so I manually deleted the stack from AWS console & redeploy the serverless again and worked perfectly.

I hope it helps you.
Should serverless have done the stack removal and re-creation for users automatically? Or at least provide this as an option?

Edit:
Interesting to see that there's #5631 that's supposed to fix this issue. Then why are we still seeing this issue?
yeah I think serverless should have a way of dealing with this. For example, 'sls remove' should just take care of it automatically
Same behavior, all these days I was resorting to deleting the stack manually and rerunning my pipeline. But, it would be very cool to stay hands-off from AWS console in ROLLBACK_COMPLETE scenarios.

+1 to have this fixed.
@cfbao & @ramgrandhi, do you happen to use a dedicated deployment bucket in your serverless.yml file? I ran into the same problem, and I was puzzled by the fact that [this commit](https://github.com/serverless/serverless/pull/5631/commits/25164d884cf86f3d62350d473011594575066241) indeed looked like it was supposed to fix it.

It turns out that serverless has a sort of "lazy" stack creation, and the code is in a different file (updateStack.js, go figure). That one gets triggered when you deploy using an existing deployment bucket instead of the default setting of letting Serverless create the bucket for you. And in that case, the stack is still created with the ROLLBACK action on failure.

The fix is just the same as #5631, I'll submit a pull request shortly.
> @cfbao & @ramgrandhi, do you happen to use a dedicated deployment bucket in your serverless.yml file?

Yes, I do! Sounds like you've figured it out. Thanks in advance.

Solution patch:
diff --git a/lib/plugins/aws/lib/updateStack.js b/lib/plugins/aws/lib/updateStack.js
index 0677d2d6336..9944587b1df 100644
--- a/lib/plugins/aws/lib/updateStack.js
+++ b/lib/plugins/aws/lib/updateStack.js
@@ -24,7 +24,7 @@ module.exports = {
 
     const params = {
       StackName: stackName,
-      OnFailure: 'ROLLBACK',
+      OnFailure: 'DELETE',
       Capabilities: ['CAPABILITY_IAM', 'CAPABILITY_NAMED_IAM'],
       Parameters: [],
       TemplateURL: templateUrl,


Test patch:
diff --git a/lib/plugins/aws/lib/updateStack.test.js b/lib/plugins/aws/lib/updateStack.test.js
index 6398d15ee33..a91465c888a 100644
--- a/lib/plugins/aws/lib/updateStack.test.js
+++ b/lib/plugins/aws/lib/updateStack.test.js
@@ -51,7 +51,7 @@ describe('updateStack', () => {
         expect(
           createStackStub.calledWithExactly('CloudFormation', 'createStack', {
             StackName: awsDeploy.provider.naming.getStackName(),
-            OnFailure: 'ROLLBACK',
+            OnFailure: 'DELETE',
             Capabilities: ['CAPABILITY_IAM', 'CAPABILITY_NAMED_IAM'],
             Parameters: [],
             TemplateURL: `https://s3.amazonaws.com/${awsDeploy.bucketName}/${awsDeploy.serverless.service.package.artifactDirectoryName}/${compiledTemplateFileName}`,