Repository: aws/aws-sam-cli

Issue:
Golang binary provided as a zip file does not work with v0.3.0
<!--

Before reporting a new issue, make sure we don't have any duplicates already open or closed by
searching the issues list. If there is a duplicate, re-open or add a comment to the
existing issue instead of creating a new one. If you are reporting a bug,
make sure to include relevant information asked below to help with debugging.

## GENERAL HELP QUESTIONS ##
Github Issues is for bug reports and feature requests. If you have general support
questions, the following locations are a good place:

- Slack channel (#samdev): https://awssamopensource.splashthat.com/
- Post a question in StackOverflow  with "aws-sam-cli" tag

-->

**Description:**

After upgrading to 0.3.0 I am now getting the following error while using `sam local start-api`

```
{
  "errorMessage": "fork/exec /var/task/main: permission denied",
  "errorType": "PathError"
}
```
I have tried rebuilding my functions using

```
	GOOS=linux go build -o main main.go
	zip -j main.zip main
```

but that doesn't seem to be fixing the issue.

**Additional environment details (Ex: Windows, Mac, Amazon Linux etc)**
Mac/OSX

**Output of `sam --version`**:
SAM CLI, version 0.3.0

**Optional Debug logs**:
```
2018-05-08 21:40:21 http://localhost:None "POST /v1.35/images/create?tag=go1.x&fromImage=lambci%2Flambda HTTP/1.1" 200 None

Fetching lambci/lambda:go1.x Docker container image......
2018-05-08 21:40:21 Mounting /private/var/folders/6k/wl5glw6x5vb2s4z2f6yfqr280000gn/T/tmplyA5JV as /var/task:ro inside runtime container
2018-05-08 21:40:21 http://localhost:None "POST /v1.35/containers/create HTTP/1.1" 201 90
2018-05-08 21:40:21 http://localhost:None "GET /v1.35/containers/e0bbd05b31ca340043f139387946e89ac1fb69d0c9b32ff19d81e456d5522bc6/json HTTP/1.1" 200 None
2018-05-08 21:40:21 http://localhost:None "GET /v1.35/containers/e0bbd05b31ca340043f139387946e89ac1fb69d0c9b32ff19d81e456d5522bc6/json HTTP/1.1" 200 None
2018-05-08 21:40:22 http://localhost:None "POST /v1.35/containers/e0bbd05b31ca340043f139387946e89ac1fb69d0c9b32ff19d81e456d5522bc6/start HTTP/1.1" 204 0
2018-05-08 21:40:22 Starting a timer for 90 seconds for function 'RegionRates'
2018-05-08 21:40:22 http://localhost:None "GET /v1.35/containers/e0bbd05b31ca340043f139387946e89ac1fb69d0c9b32ff19d81e456d5522bc6/json HTTP/1.1" 200 None
2018-05-08 21:40:22 http://localhost:None "POST /containers/e0bbd05b31ca340043f139387946e89ac1fb69d0c9b32ff19d81e456d5522bc6/attach?stream=1&stdin=0&logs=1&stderr=1&stdout=1 HTTP/1.1" 101 0
START RequestId: 9c3b6799-610b-1d1a-9e28-e37afcec3d87 Version: $LATEST
END RequestId: 9c3b6799-610b-1d1a-9e28-e37afcec3d87
REPORT RequestId: 9c3b6799-610b-1d1a-9e28-e37afcec3d87	Duration: 1.43 ms	Billed Duration: 100 ms	Memory Size: 128 MB	Max Memory Used: 6 MB
{
  "errorMessage": "fork/exec /var/task/main: permission denied",
  "errorType": "PathError"
}
2018-05-08 21:40:22 http://localhost:None "GET /v1.35/containers/e0bbd05b31ca340043f139387946e89ac1fb69d0c9b32ff19d81e456d5522bc6/json HTTP/1.1" 200 None
2018-05-08 21:40:22 http://localhost:None "DELETE /v1.35/containers/e0bbd05b31ca340043f139387946e89ac1fb69d0c9b32ff19d81e456d5522bc6?force=True&link=False&v=False HTTP/1.1" 204 0
2018-05-08 21:40:22 Function returned an invalid response (must include one of: body, headers or statusCode in the response object). Response received:
2018-05-08 21:40:22 127.0.0.1 - - [08/May/2018 21:40:22] "POST /v2/exports/rates HTTP/1.1" 502 -
```


Discussion:
You should chmod and give your binary executable permissions 
I'm also getting this on a Go project. My reproducible branch is at https://github.com/nzoschke/gofaas/pull/62
Turns out, this is a regression from previous version because Python's ZipFile module does not retain file permissions on unzipping - https://bugs.python.org/issue15795. 

Several folks have fixed it in different ways (ex: https://www.burgundywall.com/post/preserving-file-perms-with-python-zipfile-module). We need to implement a clean solution to this problem.

We will prioritize this and get a release out asap
We did some digging in on the awsdevelopers Slack. With some sleep and debug statements I see that the tmp dir doesn't have any executable bits on the main file:

```
$ cd /var/folders/px/fd8j3qvn13gcxw9_nw25pphw0000gn/T/tmpKXhUgv
$ ls -al
total 36816
drwx------    3 noah  staff       102 May  9 07:45 .
drwx------@ 259 noah  staff      8806 May  9 07:45 ..
-rw-r--r--    1 noah  staff  18846338 May  9 07:45 main
```


@sanathkr found this, which points to Python zip library:


> looks like zipfile Python module doesn’t retain permissions on unzip.. - 
> https://stackoverflow.com/questions/434641/how-do-i-set-permissions-attributes-on-a-file-in-a-zip-file-using-pythons-zip/6297838#6297838
This problems is specifically only when you bundle the Golang binary as a zipfile and provide to CodeUri. As a workaround, you can set CodeUri to be a path to a folder that contains your binary. This will work. In fact this will be faster because SAM CLI does not have to unzip every time you run the invoke.

If you have something like:

```
MyFunction:
  Type: AWS::Serverless::Function
  Properties:
     ...
     CodeUri: ./myfunction.zip
```

Instead, convert it to something like:
```
MyFunction:
  Type: AWS::Serverless::Function
  Properties:
     ...
     CodeUri: ./build
```

Where the `build` folder contains your Golang binary. This will save you the step of building a zip file and also improve performance because SAM CLI doesn't have to unzip on every invoke. 
> This problems is specifically only when you bundle the Golang binary as a zipfile and provide to CodeUri. 

I can confirm that this is an issue on Linux for a Python project (the issue is not limited to Golang on MacOS).

> As a workaround, you can set CodeUri to be a path to a folder that contains your binary. 

I just tested this, and it worked properly. Thanks!

Thanks all!
Reopening. The root cause of this is not fixed.

Resolving patch:
diff --git a/samcli/local/lambdafn/runtime.py b/samcli/local/lambdafn/runtime.py
--- a/samcli/local/lambdafn/runtime.py
+++ b/samcli/local/lambdafn/runtime.py
@@ -4,7 +4,6 @@
 
 import os
 import shutil
-import zipfile
 import tempfile
 import signal
 import logging
@@ -12,6 +11,7 @@
 from contextlib import contextmanager
 
 from samcli.local.docker.lambda_container import LambdaContainer
+from .zip import unzip
 
 LOG = logging.getLogger(__name__)
 
@@ -190,8 +190,7 @@ def _unzip_file(filepath):
 
     LOG.info("Decompressing %s", filepath)
 
-    zip_ref = zipfile.ZipFile(filepath, 'r')
-    zip_ref.extractall(temp_dir)
+    unzip(filepath, temp_dir)
 
     # The directory that Python returns might have symlinks. The Docker File sharing settings will not resolve
     # symlinks. Hence get the real path before passing to Docker.
diff --git a/samcli/local/lambdafn/zip.py b/samcli/local/lambdafn/zip.py
new file mode 100644
--- /dev/null
+++ b/samcli/local/lambdafn/zip.py
@@ -0,0 +1,58 @@
+"""
+Helper methods to unzip an archive preserving the file permissions. Python's zipfile module does not yet support
+this feature natively (https://bugs.python.org/issue15795).
+"""
+
+import os
+import zipfile
+import logging
+
+LOG = logging.getLogger(__name__)
+
+
+def unzip(zip_file_path, output_dir):
+    """
+    Unzip the given file into the given directory while preserving file permissions in the process.
+
+    Parameters
+    ----------
+    zip_file_path : str
+        Path to the zip file
+
+    output_dir : str
+        Path to the directory where the it should be unzipped to
+    """
+
+    with zipfile.ZipFile(zip_file_path, 'r') as zip_ref:
+
+        # For each item in the zip file, extract the file and set permissions if available
+        for file_info in zip_ref.infolist():
+            name = file_info.filename
+            extracted_path = os.path.join(output_dir, name)
+
+            zip_ref.extract(name, output_dir)
+            _set_permissions(file_info, extracted_path)
+
+
+def _set_permissions(zip_file_info, extracted_path):
+    """
+    Sets permissions on the extracted file by reading the ``external_attr`` property of given file info.
+
+    Parameters
+    ----------
+    zip_file_info : zipfile.ZipInfo
+        Object containing information about a file within a zip archive
+
+    extracted_path : str
+        Path where the file has been extracted to
+    """
+
+    # Permission information is stored in first two bytes.
+    permission = zip_file_info.external_attr >> 16
+    if not permission:
+        # Zips created on certain Windows machines, however, might not have any permission information on them.
+        # Skip setting a permission on these files.
+        LOG.debug("File %s in zipfile does not have permission information", zip_file_info.filename)
+        return
+
+    os.chmod(extracted_path, permission)


Test patch:
diff --git a/tests/integration/local/invoke/invoke_integ_base.py b/tests/integration/local/invoke/invoke_integ_base.py
--- a/tests/integration/local/invoke/invoke_integ_base.py
+++ b/tests/integration/local/invoke/invoke_integ_base.py
@@ -15,9 +15,10 @@ def setUpClass(cls):
 
         integration_dir = str(Path(__file__).resolve().parents[2])
 
-        cls.template_path = integration_dir + "/testdata/invoke/template.yml"
-        cls.event_path = integration_dir + "/testdata/invoke/event.json"
-        cls.env_var_path = integration_dir + "/testdata/invoke/vars.json"
+        cls.test_data_path = os.path.join(integration_dir, "testdata")
+        cls.template_path = os.path.join(cls.test_data_path, "invoke", "template.yml")
+        cls.event_path = os.path.join(cls.test_data_path, "invoke", "event.json")
+        cls.env_var_path = os.path.join(cls.test_data_path, "invoke", "vars.json")
 
     @classmethod
     def base_command(cls):
diff --git a/tests/integration/local/invoke/runtimes/__init__.py b/tests/integration/local/invoke/runtimes/__init__.py
new file mode 100644
diff --git a/tests/integration/local/invoke/runtimes/test_with_runtime_zips.py b/tests/integration/local/invoke/runtimes/test_with_runtime_zips.py
new file mode 100644
--- /dev/null
+++ b/tests/integration/local/invoke/runtimes/test_with_runtime_zips.py
@@ -0,0 +1,39 @@
+import os
+import tempfile
+
+from subprocess import Popen, PIPE
+from nose_parameterized import parameterized, param
+
+from tests.integration.local.invoke.invoke_integ_base import InvokeIntegBase
+
+
+class TestWithDifferentLambdaRuntimeZips(InvokeIntegBase):
+
+    def setUp(self):
+
+        self.template_path = os.path.join(self.test_data_path, "invoke", "runtimes", "template.yaml")
+
+        self.events_file = tempfile.NamedTemporaryFile()
+        self.events_file.write('"yolo"')  # Just empty event
+        self.events_file.flush()
+
+        self.events_file_path = self.events_file.name
+
+    def tearDown(self):
+        self.events_file.close()
+
+    @parameterized.expand([
+        param("Go1xFunction"),
+        param("Java8Function")
+    ])
+    def test_runtime_zip(self, function_name):
+        command_list = self.get_command_list(function_name,
+                                             template_path=self.template_path,
+                                             event_path=self.events_file_path)
+
+        process = Popen(command_list, stdout=PIPE)
+        return_code = process.wait()
+
+        self.assertEquals(return_code, 0)
+        process_stdout = b"".join(process.stdout.readlines()).strip()
+        self.assertEquals(process_stdout.decode('utf-8'), '"Hello World"')
diff --git a/tests/integration/testdata/invoke/runtimes/go1.x/main.go b/tests/integration/testdata/invoke/runtimes/go1.x/main.go
new file mode 100644
--- /dev/null
+++ b/tests/integration/testdata/invoke/runtimes/go1.x/main.go
@@ -0,0 +1,18 @@
+package main
+
+import (
+	"github.com/aws/aws-lambda-go/lambda"
+)
+
+// Handler is your Lambda function handler
+// It uses Amazon API Gateway request/responses provided by the aws-lambda-go/events package,
+// However you could use other event sources (S3, Kinesis etc), or JSON-decoded primitive types such as 'string'.
+func Handler() (string, error) {
+
+	return "Hello World", nil
+
+}
+
+func main() {
+	lambda.Start(Handler)
+}
diff --git a/tests/integration/testdata/invoke/runtimes/go1.x/main.zip b/tests/integration/testdata/invoke/runtimes/go1.x/main.zip
new file mode 100644
Binary files /dev/null and b/tests/integration/testdata/invoke/runtimes/go1.x/main.zip differ
diff --git a/tests/integration/testdata/invoke/runtimes/java8/pom.xml b/tests/integration/testdata/invoke/runtimes/java8/pom.xml
new file mode 100644
--- /dev/null
+++ b/tests/integration/testdata/invoke/runtimes/java8/pom.xml
@@ -0,0 +1,47 @@
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
+    <modelVersion>4.0.0</modelVersion>
+    <groupId>helloworld</groupId>
+    <artifactId>HelloWorld</artifactId>
+    <version>1.0</version>
+    <packaging>jar</packaging>
+    <name>A sample Hello World created for SAM CLI.</name>
+    <properties>
+        <maven.compiler.source>1.8</maven.compiler.source>
+        <maven.compiler.target>1.8</maven.compiler.target>
+    </properties>
+
+    <dependencies>
+        <dependency>
+            <groupId>com.amazonaws</groupId>
+            <artifactId>aws-lambda-java-core</artifactId>
+            <version>1.1.0</version>
+        </dependency>
+        <dependency>
+          <groupId>junit</groupId>
+          <artifactId>junit</artifactId>
+          <version>4.12</version>
+          <scope>test</scope>
+        </dependency>
+    </dependencies>
+
+    <build>
+      <plugins>
+        <plugin>
+          <groupId>org.apache.maven.plugins</groupId>
+          <artifactId>maven-shade-plugin</artifactId>
+          <version>3.1.1</version>
+          <configuration>
+          </configuration>
+          <executions>
+            <execution>
+              <phase>package</phase>
+              <goals>
+                <goal>shade</goal>
+              </goals>
+            </execution>
+          </executions>
+        </plugin>
+      </plugins>
+    </build>
+</project>
diff --git a/tests/integration/testdata/invoke/runtimes/java8/src/main/java/helloworld/App.java b/tests/integration/testdata/invoke/runtimes/java8/src/main/java/helloworld/App.java
new file mode 100644
--- /dev/null
+++ b/tests/integration/testdata/invoke/runtimes/java8/src/main/java/helloworld/App.java
@@ -0,0 +1,22 @@
+package helloworld;
+
+import java.io.BufferedReader;
+import java.io.InputStreamReader;
+import java.io.IOException;
+import java.net.URL;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.StringJoiner;
+
+import com.amazonaws.services.lambda.runtime.Context;
+import com.amazonaws.services.lambda.runtime.RequestHandler;
+
+/**
+ * Handler for requests to Lambda function.
+ */
+public class App implements RequestHandler<Object, Object> {
+
+    public Object handleRequest(final Object input, final Context context) {
+        return "Hello World";
+    }
+}
diff --git a/tests/integration/testdata/invoke/runtimes/template.yaml b/tests/integration/testdata/invoke/runtimes/template.yaml
new file mode 100644
--- /dev/null
+++ b/tests/integration/testdata/invoke/runtimes/template.yaml
@@ -0,0 +1,19 @@
+AWSTemplateFormatVersion : '2010-09-09'
+Transform: AWS::Serverless-2016-10-31
+Resources:
+
+  Go1xFunction:
+    Type: AWS::Serverless::Function
+    Properties:
+      Handler: main
+      Runtime: go1.x
+      CodeUri: ./go1.x/main.zip
+      Timeout: 300
+
+  Java8Function:
+    Type: AWS::Serverless::Function
+    Properties:
+      Handler: helloworld.App::handleRequest
+      Runtime: java8
+      CodeUri: ./java8/target/HelloWorld-1.0.jar
+      Timeout: 300
diff --git a/tests/unit/local/lambdafn/test_runtime.py b/tests/unit/local/lambdafn/test_runtime.py
--- a/tests/unit/local/lambdafn/test_runtime.py
+++ b/tests/unit/local/lambdafn/test_runtime.py
@@ -337,22 +337,19 @@ def test_must_return_a_valid_file(self, unzip_file_mock, shutil_mock, os_mock):
 class TestUnzipFile(TestCase):
 
     @patch("samcli.local.lambdafn.runtime.tempfile")
-    @patch("samcli.local.lambdafn.runtime.zipfile")
+    @patch("samcli.local.lambdafn.runtime.unzip")
     @patch("samcli.local.lambdafn.runtime.os")
-    def test_must_unzip(self, os_mock, zipfile_mock, tempfile_mock):
+    def test_must_unzip(self, os_mock, unzip_mock, tempfile_mock):
         inputpath = "somepath"
         tmpdir = "/tmp/dir"
         realpath = "/foo/bar/tmp/dir/code.zip"
-        zipref = Mock()
 
         tempfile_mock.mkdtemp.return_value = tmpdir
-        zipfile_mock.ZipFile.return_value = zipref
         os_mock.path.realpath.return_value = realpath
 
         output = _unzip_file(inputpath)
         self.assertEquals(output, realpath)
 
         tempfile_mock.mkdtemp.assert_called_with()
-        zipfile_mock.ZipFile.assert_called_with(inputpath, 'r')  # Open file for reading
-        zipref.extractall.assert_called_with(tmpdir)  # unzip files to temporary directory
+        unzip_mock.assert_called_with(inputpath, tmpdir)  # unzip files to temporary directory
         os_mock.path.realpath(tmpdir)  # Return the real path of temporary directory
diff --git a/tests/unit/local/lambdafn/test_zip.py b/tests/unit/local/lambdafn/test_zip.py
new file mode 100644
--- /dev/null
+++ b/tests/unit/local/lambdafn/test_zip.py
@@ -0,0 +1,79 @@
+
+import stat
+import zipfile
+import os
+import shutil
+
+from samcli.local.lambdafn.zip import unzip
+
+from tempfile import NamedTemporaryFile, mkdtemp
+from contextlib import contextmanager
+from unittest import TestCase
+from nose_parameterized import parameterized, param
+
+
+class TestUnzipWithPermissions(TestCase):
+
+    files_with_permissions = {
+        "folder1/1.txt": 0o644,
+        "folder1/2.txt": 0o777,
+        "folder2/subdir/1.txt": 0o666,
+        "folder2/subdir/2.txt": 0o400
+    }
+
+    @parameterized.expand([param(True), param(False)])
+    def test_must_unzip(self, check_permissions):
+
+        with self._create_zip(self.files_with_permissions, check_permissions) as zip_file_name:
+            with self._temp_dir() as extract_dir:
+
+                unzip(zip_file_name, extract_dir)
+
+                for root, dirs, files in os.walk(extract_dir):
+                    for file in files:
+                        filepath = os.path.join(extract_dir, root, file)
+                        perm = oct(stat.S_IMODE(os.stat(filepath).st_mode))
+                        key = os.path.relpath(filepath, extract_dir)
+                        expected_permission = oct(self.files_with_permissions[key])
+
+                        self.assertIn(key, self.files_with_permissions)
+
+                        if check_permissions:
+                            self.assertEquals(expected_permission,
+                                              perm,
+                                              "File {} has wrong permission {}".format(key, perm))
+
+    @contextmanager
+    def _create_zip(self, files_with_permissions, add_permissions=True):
+
+        zipfilename = None
+        data = b'hello world'
+        try:
+            zipfilename = NamedTemporaryFile(mode="w+b").name
+
+            zf = zipfile.ZipFile(zipfilename, "w", zipfile.ZIP_DEFLATED)
+            for filename, perm in files_with_permissions.items():
+                fileinfo = zipfile.ZipInfo(filename)
+
+                if add_permissions:
+                    fileinfo.external_attr = perm << 16
+
+                zf.writestr(fileinfo, data)
+
+            zf.close()
+
+            yield zipfilename
+
+        finally:
+            if zipfilename:
+                os.remove(zipfilename)
+
+    @contextmanager
+    def _temp_dir(self):
+        name = None
+        try:
+            name = mkdtemp()
+            yield name
+        finally:
+            if name:
+                shutil.rmtree(name)