Repository: localstack/localstack

Issue:
bug: Lambda execution doesn't work with Golang binary lambdas: <title>
### Is there an existing issue for this?

- [X] I have searched the existing issues

### Current Behavior

Running `awslocal lambda invoke --function-name "localStage-mediaStack-backfillGetFunction91FAA-b70d6394" "manual_ test_output.json"` even a single time causes localstack to continuously try and fail to run the lambda, the terminal hands whilst docker compose logs output:
```
localstack_main  | 2023-06-07T16:47:15.160  INFO --- [   asgi_gw_4] localstack.request.aws     : AWS sts.AssumeRole => 200
localstack_main  | 2023-06-07T16:47:16.000  INFO --- [   asgi_gw_0] localstack.request.http    : POST /_localstack_lambda/f1a13ad3d39e7c5f771f6cf85f78456d/status/f1a13ad3d39e7c5f771f6cf85f78456d/error => 202
localstack_main  | 2023-06-07T16:47:16.378  WARN --- [   asgi_gw_2] l.s.a.i.executor_endpoint  : Execution environment startup failed: {"errorMessage":"Error: fork/exec /var/task/bootstrap: no such file or directory","errorType":"Runtime.InvalidEntrypoint"}
```

The logs won't stop until I pull the container down and restart it (hence I know it's continuously retrying to no avail)

### Expected Behavior

Lambda should be executed just like it does on AWS, returning a sample return string like "Test" (the lambda is very simple)
Even if it does fail (I'm not sure why it'd fail here but not on AWS) then it should try a small finite number of times and then stop

### How are you starting LocalStack?

With a docker-compose file (shown below)

### Steps To Reproduce

I don't think my code itself is the problem since it does work fine on AWS, but for a bit of context:
I use the cdk to compile the binaries and infrastructure which is deployed to LocalStack via:
    cdklocal deploy -a "cdk.out/assembly-localStage/" --all --require-approval "never"

My infra:
https://github.com/KamWithK/exSTATic-backend/tree/master/infrastructure

I can potentially create a small isolated example which illustrates the problem if needed

### Environment

```markdown
version: "3.8"

services:
  localstack:
    container_name: "${LOCALSTACK_DOCKER_NAME-localstack_main}"
    image: localstack/localstack:latest
    ports:
      - "127.0.0.1:4566:4566" # LocalStack Gateway
      - "127.0.0.1:4510-4559:4510-4559" # external services port range
    environment:
      - DEBUG=${DEBUG-}
      - DOCKER_HOST=unix:///var/run/docker.sock
      - PERSISTENCE=/tmp/localstack/data
      - AWS_DEFAULT_REGION=ap-southeast-2
    volumes:
      - "${LOCALSTACK_VOLUME_DIR:-./volume}:/var/lib/localstack"
      - "/var/run/docker.sock:/var/run/docker.sock"
    healthcheck:
      test: curl http://localhost:4566/_localstack/health
      interval: 1s
      timeout: 1s
      retries: 10
```


### Anything else?

Here's some function info:
```
{
            "FunctionName": "localStage-mediaStack-backfillGetFunction91FAA-b70d6394",
            "FunctionArn": "arn:aws:lambda:us-east-1:000000000000:function:localStage-mediaStack-backfillGetFunction91FAA-b70d6394",
            "Runtime": "provided.al2",
            "Role": "arn:aws:iam::000000000000:role/localStage-mediaStack-backfillGetFunctionServi-f0162982",
            "Handler": "bootstrap",
            "CodeSize": 14604,
            "Description": "",
            "Timeout": 3,
            "MemorySize": 128,
            "LastModified": "2023-06-07T15:58:33.766076+0000",
            "CodeSha256": "nqvy9NYt9j59ura5fEUE4QaeXOTuTiEsDNgEvsSQGdk=",
            "Version": "$LATEST",
            "TracingConfig": {
                "Mode": "PassThrough"
            },
            "RevisionId": "377a6465-e32b-4ed3-a9f1-396af7ee0d86",
            "PackageType": "Zip",
            "Architectures": [
                "x86_64"
            ],
            "EphemeralStorage": {
                "Size": 512
            },
            "SnapStart": {
                "ApplyOn": "None",
                "OptimizationStatus": "Off"
            }
        }
```

Test lambda:
```golang
package main

import (
	"github.com/aws/aws-lambda-go/lambda"
)

func HandleRequest() (string, error) {
	return "Test", nil
}

func main() {
	lambda.Start(HandleRequest)
}
```


I did find this issue which sounded similar at first, but in my case the code does run on AWS (so I think it's a different problem, put here for reference though):
https://github.com/localstack/localstack/issues/4216

Any help would be greatly appreciated!


Discussion:
Welcome to LocalStack! Thanks for reporting your first issue and our team will be working towards fixing the issue for you or reach out for more background information. We recommend joining our [Slack Community](https://localstack.cloud/contact/) for real-time help and drop a message to LocalStack Pro Support if you are a Pro user! If you are willing to contribute towards fixing this issue, please have a look at our [contributing guidelines](https://github.com/localstack/.github/blob/main/CONTRIBUTING.md) and our [contributing guide](https://docs.localstack.cloud/contributing/).
Hi!
Thank you for your issue report. Can you please check (for example by running a get-function on the deployed function after deployment and downloading from the code archive url) if your deployment zip archive has a "bootstrap" file in it?
You can see here in the artifacts that there are bootstrap files per lambda:
![image](https://github.com/localstack/localstack/assets/51506271/64a4b546-6aed-41c3-8b7f-fc5eac61df8f)

The output of get function:
```
{
    "Configuration": {
        "FunctionName": "localStage-mediaStack-backfillGetFunction91FAA-b73f1ed8",
        "FunctionArn": "arn:aws:lambda:us-east-1:000000000000:function:localStage-mediaStack-backfillGetFunction91FAA-b73f1ed8",
        "Runtime": "provided.al2",
        "Role": "arn:aws:iam::000000000000:role/localStage-mediaStack-backfillGetFunctionServi-402dbbd3",
        "Handler": "bootstrap",
        "CodeSize": 14604,
        "Description": "",
        "Timeout": 3,
        "MemorySize": 128,
        "LastModified": "2023-06-07T17:16:27.953912+0000",
        "CodeSha256": "nqvy9NYt9j59ura5fEUE4QaeXOTuTiEsDNgEvsSQGdk=",
        "Version": "$LATEST",
        "TracingConfig": {
            "Mode": "PassThrough"
        },
        "RevisionId": "d7c87b6b-c1b8-4294-8b61-d00928363c33",
        "State": "Active",
        "LastUpdateStatus": "Successful",
        "PackageType": "Zip",
        "Architectures": [
        ],
        "EphemeralStorage": {
            "Size": 512
        },
        "SnapStart": {
            "ApplyOn": "None",
            "OptimizationStatus": "Off"
        },
        "RuntimeVersionConfig": {
            "RuntimeVersionArn": "arn:aws:lambda:us-east-1::runtime:8eeff65f6809a3ce81507fe733fe09b835899b99481ba22fd75b5a7338290ec1"

        }
    },
    "Code": {
        "RepositoryType": "S3",
        "Location": "http://s3.localhost.localstack.cloud:4566/awslambda-us-east-1-tasks/snapshots/000000000000/localStage-mediaStack-backfillGetFunction91FAA-b73f1ed8-38ad0c65-fc7c-4521-92c4-9283e4138165?AWSAccessKeyId=000000000000&Signature=Rq4dhGYZy5tDkMgbzoWjKTrpQ9s%3D&Expires=1686161895"
    }
}
```

When I download the "Location" file and unzip it, it does not contain a bootstrap file because it is not the binary it is a folder with all the code itself there I believe
Thanks for checking it. Can you maybe check if it is a binary on AWS?
Yeah just checked and if I do the same steps but for AWS instead of LocalStack I get a zip folder with a binary "bootstrap" in it
So it seems the deployment process is somehow different with AWS vs LS, without the bootstrap binary it won't work.

If you can, a small reproducible example would be perfect, so we can check what is behind this.
Yeah that's a bit weird
It could be due to the CDK being weird too
I'll take a look into that and will create a small isolated example soon

Thanks for the very quick response @dfangl, much appreciated!
So I managed to make some progress on this
The initial errors over there were caused by me not properly synthasizing the full cdk stack (not a LocalStack problem, something which I just never realised on AWS due to it using the pipeline which handles that itself)

After fixing that up the bootstrap file is within the lambda both on AWS and with LocalStack!
However, LocalStack still won't let me invoke the lambda unfortunately

A different error is continuously logged when I try to run the lambda through that invoke command:
```
localstack_main  | 2023-06-08T07:59:21.932  INFO --- [   asgi_gw_0] localstack.request.http    : POST /_localstack_lambda/b1c40048f71a4c715e67a9611da63142/status/b1c40048f71a4c715e67a9611da63142/error => 202
localstack_main  | 2023-06-08T07:59:22.064  WARN --- [   asgi_gw_4] l.s.a.i.executor_endpoint  : Execution environment startup failed: {"errorMessage":"Error: fork/exec /var/task/bootstrap: permission denied","errorType":"Runtime.InvalidEntrypoint"}   
localstack_main  | 2023-06-08T07:59:22.757  INFO --- [   asgi_gw_1] localstack.request.aws     : AWS sts.AssumeRole => 200
```

Once again tried to research this one but haven't had any luck
No idea why it happens with LocalStack but not AWS, since they're virtually set up the exact same way

I'm going through and creating a slightly more isolated example to illustrate the problem a bit clearer now
I've created a minimal example to illustrate the problem and hopefully find a fix:
https://github.com/KamWithK/golang_localstack_test
This does seem to be getting a different error to what I get in the full project:
```
localstack_main  | 2023-06-08T08:32:00.877  WARN --- [    Thread-5] l.s.a.i.runtime_environmen : Executor 400607dd6cc7b815cdc789eb28baea05 for function arn:aws:lambda:us-east-1:000000000000:function:testFunction:$LATEST timed out during startup
```
I did manage to find [this issue](https://github.com/localstack/localstack/issues/7792) about the timeout errors which suggested switching to legacy lambda and that just causes the permission error again:
```
localstack_main  | 2023-06-08T12:43:09.632  INFO --- [uest_thread)] l.s.a.lambda_executors     : Empty event body specified for invocation of Lambda "arn:aws:lambda:us-east-1:000000000000:function:testFunction"
localstack_main  | 2023-06-08T12:43:09.820  INFO --- [uest_thread)] l.u.container_networking   : Determined main container network: golang_localstack_test_default
localstack_main  | 2023-06-08T12:43:09.839  INFO --- [uest_thread)] l.u.container_networking   : Determined main container target IP: 172.29.0.2
localstack_main  | 2023-06-08T12:43:09.839  INFO --- [uest_thread)] l.s.a.lambda_executors     : Running lambda: arn:aws:lambda:us-east-1:000000000000:function:testFunction
localstack_main  | 2023-06-08T12:43:29.476  INFO --- [uest_thread)] l.s.a.lambda_executors     : writing log to file '/var/lib/localstack/tmp/lambda_x2is39hb.log'
localstack_main  | 2023-06-08T12:43:29.547  INFO --- [   asgi_gw_1] localstack.request.aws     : AWS cloudwatch.PutMetricData => 200
localstack_main  | 2023-06-08T12:43:29.557  INFO --- [   asgi_gw_2] localstack.request.aws     : AWS cloudwatch.PutMetricData => 200
localstack_main  | 2023-06-08T12:43:29.564  INFO --- [uest_thread)] l.s.awslambda.lambda_api   : Error executing Lambda function arn:aws:lambda:us-east-1:000000000000:function:testFunction: Lambda process returned with error. Result: {"errorType":"exitError","errorMessage":"RequestId: 4f313260-b680-1f71-552e-fac2d03f81c3 Error: fork/exec /var/task/bootstrap: permission denied"}. Output:   
localstack_main  | START RequestId: 4f313260-b680-1f71-552e-fac2d03f81c3 Version: $LATEST
localstack_main  | END RequestId: 4f313260-b680-1f71-552e-fac2d03f81c3
localstack_main  | REPORT RequestId: 4f313260-b680-1f71-552e-fac2d03f81c3       Init Duration: 42.42 ms   Duration: 3.31 ms       Billed Duration: 4 ms   Memory Size: 1536 MB    Max Memory Used: 10 MB     Traceback (most recent call last):
localstack_main  |   File "/opt/code/localstack/localstack/services/awslambda/lambda_executors.py", line 667, in run_lambda_executor
localstack_main  |     result, log_output = self.execute_in_container(
localstack_main  |   File "/opt/code/localstack/localstack/services/awslambda/lambda_executors.py", line 1298, in execute_in_container
localstack_main  |     return DOCKER_CLIENT.start_container(
localstack_main  |   File "/opt/code/localstack/localstack/utils/container_utils/docker_sdk_client.py", line 562, in start_container
localstack_main  |     raise ContainerException(
localstack_main  | localstack.utils.container_utils.container_client.ContainerException: Docker container returned with exit code 1
localstack_main  |
localstack_main  | The above exception was the direct cause of the following exception:
localstack_main  |
localstack_main  | Traceback (most recent call last):
localstack_main  |   File "/opt/code/localstack/localstack/services/awslambda/lambda_api.py", line 463, in run_lambda
localstack_main  |     result = LAMBDA_EXECUTOR.execute(
localstack_main  |   File "/opt/code/localstack/localstack/services/awslambda/lambda_executors.py", line 494, in execute
localstack_main  |     return do_execute()
localstack_main  |   File "/opt/code/localstack/localstack/services/awslambda/lambda_executors.py", line 484, in do_execute
localstack_main  |     return _run(func_arn=func_arn)
localstack_main  |   File "/opt/code/localstack/localstack/utils/cloudwatch/cloudwatch_util.py", line 182, in wrapped
localstack_main  |   File "/opt/code/localstack/localstack/services/awslambda/lambda_executors.py", line 790, in _execute
localstack_main  |     result = self.run_lambda_executor(lambda_function=lambda_function, inv_context=inv_context)
localstack_main  |   File "/opt/code/localstack/localstack/services/awslambda/lambda_executors.py", line 712, in run_lambda_executor
localstack_main  |     raise InvocationException(
localstack_main  | localstack.services.awslambda.lambda_executors.InvocationException: Lambda process returned with error. Result: {"errorType":"exitError","errorMessage":"RequestId: 4f313260-b680-1f71-552e-fac2d03f81c3 Error: fork/exec /var/task/bootstrap: permission denied"}. Output:
localstack_main  | START RequestId: 4f313260-b680-1f71-552e-fac2d03f81c3 Version: $LATEST
localstack_main  | END RequestId: 4f313260-b680-1f71-552e-fac2d03f81c3
localstack_main  | REPORT RequestId: 4f313260-b680-1f71-552e-fac2d03f81c3       Init Duration: 42.42 ms   Duration: 3.31 ms       Billed Duration: 4 ms   Memory Size: 1536 MB    Max Memory Used: 10 MB  
localstack_main  |
localstack_main  | 2023-06-08T12:43:29.566  INFO --- [   asgi_gw_0] localstack.request.aws     : AWS lambda.Invoke => 200
```
> So it seems the deployment process is somehow different with AWS vs LS, without the bootstrap binary it won't work.
> 
> If you can, a small reproducible example would be perfect, so we can check what is behind this.

@dfangl When you have the time let me know what you think about the reproducible example I created
I tried a few more times to find and fix the problem myself but I'm stumped about what's actually going wrong
Had a very similar issue with our team today trying to deploy a golang lambda to localstack locally

We got permissiondenied as above and also runtime error 

a few things worked for us :  
-  roll back to 2.1.0 (we did this, now that things are working, I will try latest and see if everything works)
- ensure to do chmod on the bootstrap and compiled binary 
- ensure you are building with CGO_ENABLED=0 and GOOS=linux 
> Had a very similar issue with our team today trying to deploy a golang lambda to localstack locally
> 
> We got permissiondenied as above and also runtime error
> 
> a few things worked for us :
> 
>     * roll back to 2.1.0 (we did this, now that things are working, I will try latest and see if everything works)
> 
>     * ensure to do chmod on the bootstrap and compiled binary
> 
>     * ensure you are building with CGO_ENABLED=0 and GOOS=linux

I will try using version 2.1.0 of LocalStack (I assume that's what you mean) but in terms of the other tips I don't have any control over that since it's managed by one of the CDK JavaScript libraries (which should be working fine since it does work with AWS)
Maybe your first tip will work though so I'll give it a try soon!
Thank you both. I do not think there were many changes post 2.1 regarding this logic, but it does not hurt to try.

It's possible that we have to make the bootstrap file executable on our side, we will try to reproduce and get back to this.
> Had a very similar issue with our team today trying to deploy a golang lambda to localstack locally
> 
>     * roll back to 2.1.0 (we did this, now that things are working, I will try latest and see if everything works)

I did give the old version 2.1.0 a shot, but it seems like nothing has actually changed, I'm still getting the exact same permission denied errors
> Thank you both. I do not think there were many changes post 2.1 regarding this logic, but it does not hurt to try.
> 
> It's possible that we have to make the bootstrap file executable on our side, we will try to reproduce and get back to this.

Just wondering whether you managed to take a look at the example and maybe find anything going wrong?
The permission denied error specifically was fixed by doing `sudo chmod -R
a+rwx` on all artefacts that we zip up to the lambda.
it would be good to test that your side as well.

On Thu, Jun 15, 2023 at 8:43 AM Kamron Bhavnagri ***@***.***>
wrote:

> Thank you both. I do not think there were many changes post 2.1 regarding
> this logic, but it does not hurt to try.
>
> It's possible that we have to make the bootstrap file executable on our
> side, we will try to reproduce and get back to this.
>
> Just wondering whether you managed to take a look at the example and maybe
> find anything going wrong?
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/localstack/localstack/issues/8459#issuecomment-1592528806>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AF6GENTTRLSKOM7BWOTINGLXLK4J7ANCNFSM6AAAAAAY6EURYQ>
> .
> You are receiving this because you commented.Message ID:
> ***@***.***>
>

> The permission denied error specifically was fixed by doing `sudo chmod -R a+rwx` on all artefacts that we zip up to the lambda. it would be good to test that your side as well.
> […](#)
> On Thu, Jun 15, 2023 at 8:43 AM Kamron Bhavnagri ***@***.***> wrote: Thank you both. I do not think there were many changes post 2.1 regarding this logic, but it does not hurt to try. It's possible that we have to make the bootstrap file executable on our side, we will try to reproduce and get back to this. Just wondering whether you managed to take a look at the example and maybe find anything going wrong? — Reply to this email directly, view it on GitHub <[#8459 (comment)](https://github.com/localstack/localstack/issues/8459#issuecomment-1592528806)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AF6GENTTRLSKOM7BWOTINGLXLK4J7ANCNFSM6AAAAAAY6EURYQ> . You are receiving this because you commented.Message ID: ***@***.***>

The whole golang lambda function build process for me is handled by the [AWS TypeScript library](https://docs.aws.amazon.com/cdk/api/v2/docs/aws-lambda-go-alpha-readme.html) though, so I'm not sure how/where to do that exactly. How did you go about fixing it in [my example](https://github.com/KamWithK/golang_localstack_test)? If this extra step needs to happen on LocalStack but not in AWS itself, maybe LocalStack can run the command itself?
I checked out your example repository.

For me, it builds the bootstrap file with executable permissions, but this might be OS specific, can you share what OS you are on?

I have my suspicions this might fail on Windows or maybe even MacOS.

The file I download from LS via the presigned URL again, looks like this:

```
Archive:  testFunction-cba344df-5f47-4f74-9392-28561a88553a
Zip file size: 5440988 bytes, number of entries: 1
-rwxr-xr-x  4.5 unx  9345131 bl defN 80-Jan-01 00:00 bootstrap
1 file, 9345131 bytes uncompressed, 5440856 bytes compressed:  41.8%
```

I will try the behavior if I indeed create a file without the permissions set, and report back.
I can confirm that the execute permissions are not necessary on AWS, but on LocalStack.
I will try to have a fix ready today.
> I checked out your example repository.
> 
> For me, it builds the bootstrap file with executable permissions, but this might be OS specific, can you share what OS you are on?
> 
> I have my suspicions this might fail on Windows or maybe even MacOS.
> 
> The file I download from LS via the presigned URL again, looks like this:
> 
> ```
> Archive:  testFunction-cba344df-5f47-4f74-9392-28561a88553a
> Zip file size: 5440988 bytes, number of entries: 1
> -rwxr-xr-x  4.5 unx  9345131 bl defN 80-Jan-01 00:00 bootstrap
> 1 file, 9345131 bytes uncompressed, 5440856 bytes compressed:  41.8%
> ```
> 
> I will try the behavior if I indeed create a file without the permissions set, and report back.

Hey thanks for taking a look!
I am on Windows
> I can confirm that the execute permissions are not necessary on AWS, but on LocalStack. I will try to have a fix ready today.

Sounds good thanks!
> I can confirm that the execute permissions are not necessary on AWS, but on LocalStack. I will try to have a fix ready today.

Any progress @dfangl? When do you think this will be fixed and released?
I don't think chmod exists on Windows, so until this is patched I can't think of any ways to use LocalStack with Go executables on Windows

Resolving patch:
diff --git a/localstack/services/awslambda/packages.py b/localstack/services/awslambda/packages.py
--- a/localstack/services/awslambda/packages.py
+++ b/localstack/services/awslambda/packages.py
@@ -10,7 +10,7 @@
 
 LAMBDA_RUNTIME_INIT_URL = "https://github.com/localstack/lambda-runtime-init/releases/download/{version}/aws-lambda-rie-{arch}"
 
-LAMBDA_RUNTIME_DEFAULT_VERSION = "v0.1.18-pre"
+LAMBDA_RUNTIME_DEFAULT_VERSION = "v0.1.19-pre"
 LAMBDA_RUNTIME_VERSION = config.LAMBDA_INIT_RELEASE_VERSION or LAMBDA_RUNTIME_DEFAULT_VERSION
 
 # GO Lambda runtime


Test patch: