Repository: localstack/localstack

Issue:
bug: Long Running Lambda Fails StepFunction State Machine Execution
### Is there an existing issue for this?

- [X] I have searched the existing issues

### Current Behavior

As of `v3.0.0` and `v3.0.1`, StepFunction StateMachines that have long-running Lambda tasks fail execution. It also looks like the StateMachine then retries by re-invoking the lambda 3 times in the background with a 1 minute gap in between invocations. Unfortunately, the state machine will have already failed execution by this point and these lambda runs fail when they try to update the state.

The lambda is started successfully, but then fails with a timeout after 3 seconds:

```
2023-11-24T22:09:56.758 ERROR --- [ad-35 (eval)] l.s.s.a.c.eval_component   : Exception=FailureEventException, Error=Exception, Details={"taskFailedEventDetails": {"error": "Exception", "cause": "{\"errorMessage\":\"2023-11-24T22:09:56Z dbd4767f-32b8-46b7-9ef4-382ee583ad0a Task timed out after 3.00 seconds\"}", "resource": "invoke", "resourceType": "lambda"}} at '(StateTaskServiceLambda| {'comment': None, 'input_path': (InputPath| {'input_path_src': '$'}, 'output_path': (OutputPath| {'output_path': '$'}, 'state_entered_event_type': 'TaskStateEntered', 'state_exited_event_type': 'TaskStateExited', 'result_path': None, 'result_selector': None, 'retry': (RetryDecl| {'retriers': [(RetrierDecl| {'error_equals': (ErrorEqualsDecl| {'error_names': [(CustomErrorName| {'error_name': 'Lambda.ClientExecutionTimeoutException'}, (CustomErrorName| {'error_name': 'Lambda.ServiceException'}, (CustomErrorName| {'error_name': 'Lambda.AWSLambdaException'}, (CustomErrorName| {'error_name': 'Lambda.SdkClientException'}]}, 'interval_seconds': (IntervalSecondsDecl| {'seconds': 2}, 'max_attempts': (MaxAttemptsDecl| {'attempts': 6}, 'backoff_rate': (BackoffRateDecl| {'rate': 2.0}, '_attempts_counter': 0, '_next_interval_seconds': 2}]}, 'catch': None, 'timeout': (TimeoutSeconds| {'timeout_seconds': 99999999, 'is_default': None}, 'heartbeat': None, 'parameters': (Parameters| {'payload_tmpl': (PayloadTmpl| {'payload_bindings': [(PayloadBindingValue| {'field': 'FunctionName', 'value': (PayloadValueStr| {'val': 'arn:aws:lambda:us-east-1:000000000000:function:TestAppStack-lambdaslongrunning51EEA4-b04d9aee'}}, (PayloadBindingPath| {'field': 'Payload', 'path': '$'}]}}, 'name': 'long-running-task', 'state_type': <StateType.Task: 15>, 'continue_with': <localstack.services.stepfunctions.asl.component.state.state_continue_with.ContinueWithEnd object at 0xfffee6793b90>, 'resource': (ServiceResource| {'_region': '', '_account': '', 'resource_arn': 'arn:aws:states:::lambda:invoke', 'partition': 'aws', 'service_name': 'lambda', 'api_name': 'lambda', 'api_action': 'invoke', 'condition': None}}'
```

Even if I specify long timeouts on both the Lambda and the LambdaTask the state machine still fails the task after 3 seconds. This was working in version 2, and if I use the old StepFunctions provider, the StateMachine completes successfully.

### Expected Behavior

The State Machine should finish successfully because the long running lambda finishes before the timeout.

### How are you starting LocalStack?

With a docker-compose file

### Steps To Reproduce

#### How are you starting localstack (e.g., `bin/localstack` command, arguments, or `docker-compose.yml`)

I've created a repository that demonstrates the bug: https://github.com/noseworthy/localstack-sfn-bugs. I'm using localstack pro, so your terminal must have `LOCALSTACK_AUTH_TOKEN` specified. This should work with non-pro localstack however. You just need to modify the `compose.yaml` file.

1. Start localstack using docker-compose:  `docker compose up --force-recreate --build -d`

#### Client commands (e.g., AWS SDK code snippet, or sequence of "awslocal" commands)

1. Install dependencies: `yarn install`
2. Bootstrap the CDK project: `yarn cdklocal bootstrap`
3. Deploy the CDK project: `yarn cdklocal deploy`
4. Trigger the state machine: `yarn trigger`

Watch as the statemachine tries to execute, but fails saying that the long running lambda timed out after 3.00 seconds.

### Environment

```markdown
- OS: macOS Sonoma 14.1.1 (23B81)
- LocalStack: v3.0.1 Pro Docker Image
```


### Anything else?

Demo Repository: https://github.com/noseworthy/localstack-sfn-bugs


Discussion:
Welcome to LocalStack! Thanks for reporting your first issue and our team will be working towards fixing the issue for you or reach out for more background information. We recommend joining our [Slack Community](https://localstack.cloud/contact/) for real-time help and drop a message to LocalStack Pro Support if you are a Pro user! If you are willing to contribute towards fixing this issue, please have a look at our [contributing guidelines](https://github.com/localstack/.github/blob/main/CONTRIBUTING.md) and our [contributing guide](https://docs.localstack.cloud/contributing/).

Resolving patch:
diff --git a/localstack/services/stepfunctions/asl/utils/boto_client.py b/localstack/services/stepfunctions/asl/utils/boto_client.py
--- a/localstack/services/stepfunctions/asl/utils/boto_client.py
+++ b/localstack/services/stepfunctions/asl/utils/boto_client.py
@@ -2,6 +2,7 @@
 from botocore.config import Config
 
 from localstack.aws.connect import connect_to
+from localstack.services.stepfunctions.asl.component.common.timeouts.timeout import TimeoutSeconds
 
 
 def boto_client_for(region: str, account: str, service: str) -> BaseClient:
@@ -9,5 +10,10 @@ def boto_client_for(region: str, account: str, service: str) -> BaseClient:
         aws_access_key_id=account,
         region_name=region,
         service_name=service,
-        config=Config(parameter_validation=False),
+        config=Config(
+            parameter_validation=False,
+            retries={"max_attempts": 0, "total_max_attempts": 1},
+            connect_timeout=TimeoutSeconds.DEFAULT_TIMEOUT_SECONDS,
+            read_timeout=TimeoutSeconds.DEFAULT_TIMEOUT_SECONDS,
+        ),
     )


Test patch: