Repository: localstack/localstack
Pull request: Fix: CFn repeated deployments breaking
<!-- Please refer to the contribution guidelines before raising a PR: https://github.com/localstack/localstack/blob/master/docs/CONTRIBUTING.md -->

<!-- Why am I raising this PR? Add context such as related issues, PRs, or documentation. -->
## Motivation

As reported in https://github.com/localstack/localstack/issues/11333, performing a deployment of a previously deleted stack fails with the error message:

```
InvalidChangeSetStatus: ChangeSet [arn:aws:cloudformation:us-east-1:000000000000:changeSet/cdk-deploy-change-set/87846077] cannot be executed in its current status of [CREATE_COMPLETE]
```

This is because we don't filter out deleted changesets when looking for a changeset, and therefore assume the changeset already exists and incorrectly fail. Also this only happens when specifying changeset _names_, which CDK does.

This PR superscedes https://github.com/localstack/localstack/pull/11384, however @migraf wrote the initial implementation.

Closes #11333

<!-- What changes does this PR make? How does LocalStack behave differently now? -->
## Changes

- Fetch only active changesets
- Add validated test that performs the deploy/delete/deploy process

<!-- Optional section: How to test these changes? -->
<!--
## Testing

-->

<!-- Optional section: What's left to do before it can be merged? -->
<!--
## TODO

What's left to do:

- [ ] ...
- [ ] ...
-->

Problem statement:
bug: repeated CDK deploy fails with InvalidChangeSetStatus
### Is there an existing issue for this?

- [X] I have searched the existing issues

### Current Behavior

When 
I run the following commands:
* `cdklocal bootstrap`
* `cdklocal deploy --require-approval never`
* `cdklocal destroy -f`
* `cdklocal
*  deploy --require-approval never` (the same as the second command)

CDK(local) fails to deploy the stack with the following error message:

```
 ❌  CdkStack failed: InvalidChangeSetStatus: ChangeSet [arn:aws:cloudformation:us-east-1:000000000000:changeSet/cdk-deploy-change-set/87846077] cannot be executed in its current status of [CREATE_COMPLETE]
    at Request.extractError (/Users/simon/work/localstack/cdk/node_modules/.pnpm/aws-cdk@2.151.0/node_modules/aws-cdk/lib/index.js:396:46723)
    at Request.callListeners (/Users/simon/work/localstack/cdk/node_modules/.pnpm/aws-cdk@2.151.0/node_modules/aws-cdk/lib/index.js:396:91777)
    at Request.emit (/Users/simon/work/localstack/cdk/node_modules/.pnpm/aws-cdk@2.151.0/node_modules/aws-cdk/lib/index.js:396:91225)
    at Request.emit (/Users/simon/work/localstack/cdk/node_modules/.pnpm/aws-cdk@2.151.0/node_modules/aws-cdk/lib/index.js:396:199828)
    at Request.transition (/Users/simon/work/localstack/cdk/node_modules/.pnpm/aws-cdk@2.151.0/node_modules/aws-cdk/lib/index.js:396:193380)
    at AcceptorStateMachine.runTo (/Users/simon/work/localstack/cdk/node_modules/.pnpm/aws-cdk@2.151.0/node_modules/aws-cdk/lib/index.js:396:158252)
    at /Users/simon/work/localstack/cdk/node_modules/.pnpm/aws-cdk@2.151.0/node_modules/aws-cdk/lib/index.js:396:158582
    at Request.<anonymous> (/Users/simon/work/localstack/cdk/node_modules/.pnpm/aws-cdk@2.151.0/node_modules/aws-cdk/lib/index.js:396:193672)
    at Request.<anonymous> (/Users/simon/work/localstack/cdk/node_modules/.pnpm/aws-cdk@2.151.0/node_modules/aws-cdk/lib/index.js:396:199903)
    at Request.callListeners (/Users/simon/work/localstack/cdk/node_modules/.pnpm/aws-cdk@2.151.0/node_modules/aws-cdk/lib/index.js:396:91945) {
  code: 'InvalidChangeSetStatus',
  time: 2024-08-09T10:41:28.662Z,
  requestId: undefined,
  statusCode: 400,
  retryable: false,
  retryDelay: 970.1237437233376
}
```

This workflow would be very useful when working with CloudFormation as I would not need to restart LocalStack to update my stack, which requires bootstrapping again.

### Expected Behavior

The stack deploys correctly the second time to AWS.

### How are you starting LocalStack?

With the `localstack` script

### Steps To Reproduce

#### How are you starting localstack (e.g., `bin/localstack` command, arguments, or `docker-compose.yml`)

    localstack start

#### Client commands (e.g., AWS SDK code snippet, or sequence of "awslocal" commands)

    npx cdk init -l typescript -t app
    npm install --save-dev aws-cdk-local
    npx cdklocal bootstrap
    npx cdklocal deploy --require-approval never
    npx cdklocal destroy -f
    npx cdklocal deploy --require-approval never

### Environment

```markdown
- OS: macOS 14.6
- LocalStack:
  LocalStack version: 3.6.1.dev
  LocalStack build date: 2024-08-09
  LocalStack build git hash: f497a6e66
  LocalStack Docker image sha: localstack/localstack@sha256:61413db7a8b511a99bb1470e78032890c9c21ef68f0814a6f002b6a6908a747c
```


### Anything else?

_No response_

Hints from issue discussion:

Commit message: Add skipped failling test

Patch to review:
diff --git a/localstack-core/localstack/services/cloudformation/provider.py b/localstack-core/localstack/services/cloudformation/provider.py
index c7dcfcb0889db..b10617ed92ef5 100644
--- a/localstack-core/localstack/services/cloudformation/provider.py
+++ b/localstack-core/localstack/services/cloudformation/provider.py
@@ -850,7 +850,11 @@ def execute_change_set(
         **kwargs,
     ) -> ExecuteChangeSetOutput:
         change_set = find_change_set(
-            context.account_id, context.region, change_set_name, stack_name=stack_name
+            context.account_id,
+            context.region,
+            change_set_name,
+            stack_name=stack_name,
+            active_only=True,
         )
         if not change_set:
             raise ChangeSetNotFoundException(f"ChangeSet [{change_set_name}] does not exist")
diff --git a/localstack-core/localstack/services/cloudformation/stores.py b/localstack-core/localstack/services/cloudformation/stores.py
index e14a911183e54..04cc9a7a6286b 100644
--- a/localstack-core/localstack/services/cloudformation/stores.py
+++ b/localstack-core/localstack/services/cloudformation/stores.py
@@ -103,10 +103,16 @@ def find_active_stack_by_name_or_id(
 
 
 def find_change_set(
-    account_id: str, region_name: str, cs_name: str, stack_name: Optional[str] = None
+    account_id: str,
+    region_name: str,
+    cs_name: str,
+    stack_name: Optional[str] = None,
+    active_only: bool = False,
 ) -> Optional[StackChangeSet]:
     store = get_cloudformation_store(account_id, region_name)
     for stack in store.stacks.values():
+        if active_only and stack.status == "DELETE_COMPLETE":
+            continue
         if stack_name in (stack.stack_name, stack.stack_id, None):
             for change_set in stack.change_sets:
                 if cs_name in (change_set.change_set_id, change_set.change_set_name):
diff --git a/tests/aws/services/cloudformation/api/test_changesets.py b/tests/aws/services/cloudformation/api/test_changesets.py
index 57eb4e94546b9..212eeb2b799de 100644
--- a/tests/aws/services/cloudformation/api/test_changesets.py
+++ b/tests/aws/services/cloudformation/api/test_changesets.py
@@ -431,6 +431,50 @@ def test_delete_change_set_exception(snapshot, aws_client):
     snapshot.match("e2", e2)
 
 
+@markers.aws.validated
+def test_create_delete_create(aws_client, cleanups, deploy_cfn_template):
+    # create stack
+    stack_name = f"stack-{short_uid()}"
+    change_set_name = f"cs-{short_uid()}"
+
+    template_path = os.path.join(
+        os.path.dirname(__file__), "../../../templates/sns_topic_simple.yaml"
+    )
+    with open(template_path) as infile:
+        template = infile.read()
+
+    # custom cloudformation deploy process since our `deploy_cfn_template` is too smart and uses IDs, unlike the CDK
+    def deploy():
+        client = aws_client.cloudformation
+        client.create_change_set(
+            StackName=stack_name,
+            TemplateBody=template,
+            ChangeSetName=change_set_name,
+            ChangeSetType="CREATE",
+        )
+        client.get_waiter("change_set_create_complete").wait(
+            StackName=stack_name, ChangeSetName=change_set_name
+        )
+
+        client.execute_change_set(StackName=stack_name, ChangeSetName=change_set_name)
+        client.get_waiter("stack_create_complete").wait(
+            StackName=stack_name,
+        )
+
+    def delete(suppress_exception: bool = False):
+        try:
+            aws_client.cloudformation.delete_stack(StackName=stack_name)
+            aws_client.cloudformation.get_waiter("stack_delete_complete").wait(StackName=stack_name)
+        except Exception:
+            if not suppress_exception:
+                raise
+
+    deploy()
+    cleanups.append(lambda: delete(suppress_exception=True))
+    delete()
+    deploy()
+
+
 @markers.aws.validated
 def test_create_and_then_remove_non_supported_resource_change_set(deploy_cfn_template):
     # first deploy cfn with a CodeArtifact resource that is not actually supported
diff --git a/tests/aws/services/cloudformation/api/test_changesets.validation.json b/tests/aws/services/cloudformation/api/test_changesets.validation.json
index 70b374a8181f3..3efd7f491e9b3 100644
--- a/tests/aws/services/cloudformation/api/test_changesets.validation.json
+++ b/tests/aws/services/cloudformation/api/test_changesets.validation.json
@@ -5,6 +5,9 @@
   "tests/aws/services/cloudformation/api/test_changesets.py::test_create_changeset_with_stack_id": {
     "last_validated_date": "2023-11-28T06:48:23+00:00"
   },
+  "tests/aws/services/cloudformation/api/test_changesets.py::test_create_delete_create": {
+    "last_validated_date": "2024-08-13T10:46:31+00:00"
+  },
   "tests/aws/services/cloudformation/api/test_changesets.py::test_create_while_in_review": {
     "last_validated_date": "2023-11-22T07:49:15+00:00"
   },


Code review comments:
diff_hunk: @@ -103,10 +103,16 @@ def find_active_stack_by_name_or_id(
 
 
 def find_change_set(
-    account_id: str, region_name: str, cs_name: str, stack_name: Optional[str] = None
+    account_id: str,
+    region_name: str,
+    cs_name: str,
+    stack_name: Optional[str] = None,
+    active_only: bool = False,
 ) -> Optional[StackChangeSet]:
     store = get_cloudformation_store(account_id, region_name)
     for stack in store.stacks.values():
+        if active_only and stack.status == "DELETE_COMPLETE":
path: localstack-core/localstack/services/cloudformation/stores.py
text: @user1:
nit: could also reference the value from the generated API here