Repository: getmoto/moto

Issue:
Allow lambda docker volume to be configurable to avoid pipeline errors
Consider the following test code that will create a mock lambda that just returns some arbitrary value:


```
def get_mock_lambda(return_value: str):
    pfunc = """
def lambda_handler(event, context):    
    return {}
""".format(return_value.replace("\n", " "))

    zip_output = io.BytesIO()
    zip_file = zipfile.ZipFile(zip_output, "w", zipfile.ZIP_DEFLATED)
    zip_file.writestr("lambda_function.py", pfunc)
    zip_file.close()
    zip_output.seek(0)
    return zip_output.read()

def mock_lambda_call(return_value: str):
    conn = boto3.client("lambda", region_name="us-east-1")
    conn.create_function(
        FunctionName=os.environ.get('my_mock_lambda'),
        Runtime="python3.8",
        Role=get_role_name(),
        Handler="lambda_function.lambda_handler",
        Code={
            "ZipFile": get_mock_lambda(
                return_value
            )
        },
        Description="test lambda function",
        Timeout=3,
        MemorySize=128,
        Publish=True,
    )
```

I expect this to just work; in other, words no errors


When tests using this code snippet are run within a bitbucket pipeline, the following error is encountered for each test:

```
time="2022-05-31T20:36:10Z" level=info msg="Pipelines plugin request authorization." allowed=true method=GET plugin=pipelines uri=/version
time="2022-05-31T20:36:10Z" level=info msg="Pipelines plugin request authorization." allowed=true method=GET plugin=pipelines uri=/v1.41/volumes
time="2022-05-31T20:36:10Z" level=info msg="Pipelines plugin request authorization." allowed=true method=POST plugin=pipelines uri=/v1.41/volumes/create
time="2022-05-31T20:36:10Z" level=info msg="Pipelines plugin request authorization." allowed=true method=POST plugin=pipelines uri="/v1.41/images/create?tag=latest&[fromImage=docker.io](http://fromimage%3Ddocker.io/)%2Flibrary%2Falpine"
time="2022-05-31T20:36:11Z" level=info msg="Pipelines plugin request authorization." allowed=true method=GET plugin=pipelines uri="/v1.41/images/docker.io/library/alpine:latest/json"
time="2022-05-31T20:36:11Z" level=info msg="Container create request." ArgsEscaped=false AttachStderr=false AttachStdin=false AttachStdout=false ExposedPorts="map[]" Healthcheck="<nil>" Labels="map[]" MacAddress= NetworkDisabled=false OnBuild="[]" OpenStdin=false StdinOnce=false StopSignal= StopTimeout="<nil>" Tty=false plugin=pipelines
time="2022-05-31T20:36:11Z" level=info msg="Container create request." AutoRemove=false BlkioDeviceReadBps="[]" BlkioDeviceReadIOps="[]" BlkioDeviceWriteBps="[]" BlkioDeviceWriteIOps="[]" BlkioWeight=0 BlkioWeightDevice="[]" CPUCount=0 CPUPercent=0 CPUPeriod=0 CPUQuota=0 CPURealtimePeriod=0 CPURealtimeRuntime=0 CPUShares=0 CapAdd="[]" CapDrop="[]" Cgroup= CgroupParent= ConsoleSize="[0 0]" ContainerIDFile= CpusetCpus= CpusetMems= DNS="[]" DNSOptions="[]" DNSSearch="[]" DeviceCgroupRules="[]" Devices="[]" ExtraHosts="[]" GroupAdd="[]" IOMaximumBandwidth=0 IOMaximumIOps=0 Init="<nil>" IpcMode= Isolations= KernelMemory=0 Links="[]" LogConfig="{ map[]}" MaskedPaths="[]" Memory=0 MemoryReservation=0 MemorySwap=0 MemorySwappiness="<nil>" Mounts="[]" NanoCPUs=0 NetworkMode=default OomKillDisable="<nil>" OomScoreAdj=0 PidMode= PidsLimit="<nil>" PortBindings="map[]" Privileged=false PublishAllPorts=false ReadOnlyPaths="[]" RestartPolicy="{ 0}" Runtime= SecurityOpt="[]" ShmSize=0 StorageOpt="map[]" Sysctls="map[]" Ulimits="[]" UsernsMode= VolumeDriver= VolumesFrom="[]" plugin=pipelines
time="2022-05-31T20:36:11Z" level=info msg="Pipelines plugin request authorization." allowed=false method=POST plugin=pipelines uri=/v1.41/containers/create
time="2022-05-31T20:36:11.486112456Z" level=error msg="AuthZRequest for POST /v1.41/containers/create returned error: authorization denied by plugin pipelines: -v only supports $BITBUCKET_CLONE_DIR and its subdirectories"
```

The meat of it: `authorization denied by plugin pipelines: -v only supports $BITBUCKET_CLONE_DIR and its subdirectories`


This is happening because this environment does not allow volumes to be mounted outside of the source directory. However, currently within moto, volume mount points are _hard coded_:

```
            # It doesn't exist so we need to create it
            self._vol_ref.volume = self._lambda_func.docker_client.volumes.create(
                self._lambda_func.code_digest
            )
            if docker_3:
                volumes = {self.name: {"bind": "/tmp/data", "mode": "rw"}}
            else:
                volumes = {self.name: "/tmp/data"}

            self._lambda_func.docker_client.images.pull(
                ":".join(parse_image_ref("alpine"))
            )
            container = self._lambda_func.docker_client.containers.run(
                "alpine", "sleep 100", volumes=volumes, detach=True
            )
```


This problem can be easily resolved if we had the power to configure the volume bind point to be something other than `/tmp/data`.


We are using boto3 v1.16.* (python mocks)


Discussion:
Hi @dmarra, thanks for raising this! Marking it as an enhancement to make this configurable.

If you want to contribute a fix yourself, PR's are always welcome - let me know if you need some pointers to get started.
@bblommers I would be interested in taking a stab at this. I have read the contribution guidelines, but not sure how to create a PR for this project. should I be branching from master, or forking the entire project?
You'll have to fork the project @dmarra, create a branch on your own fork and then create a PR against that branch.
@bblommers , it's been a while, but I have made an attempt at this. I have a question for you though. Due to the nature of the change, I had to make some changes to one of the github actions in order to test my work (as is done for the other, similar tests). I want to avoid running the actions in my fork for this one change, so I was wondering if you had a method for local development to test this kind of stuff. 

I can cook something up myself with docker to validate my work before submitting a PR, but I figured I would ask before I do that just in case I would be re-inventing the wheel.  :D
Hey @dmarra, no, I didn't have anything smart set up - IIRC I just used the steps from the GA locally.

Are you worried about cost? As it should be free to run, considering it's open source. The Docker Tests run on their own, and it's really quick (as opposed to the full test suite, which will take a while).

Alternatively, you could submit a PR anyway, and (after I approve it) it would run the GitHub Action in our fork.
It's not so much the costs, as it's wanting to perform a more targeted test locally before I push

I think I can just stand up a docker container locally and cook up a small local test. I'd rather catch a mistake now before wasting anyone's time with a PR that might or might not be broken.  :D 

Thanks!
Hi @dmarra, did you get any progress on this? I'd be happy to take it on if you're stuck or don't have the time, just checking in to make sure I'm not stepping on your toes!
@bblommers Unfortunately I did not. I bit off a bit more than I could chew, and ended up down a rabbit hole I didn't have time to get out of :(

We have since migrated our code to GitHub, so this need became less of an issue for us. I can push what I have so far if you feel it would be valuable. I am happy to do so
That'd be useful @dmarra! I'll have a look and see if I can build on top of that.

Resolving patch:
diff --git a/moto/awslambda/models.py b/moto/awslambda/models.py
--- a/moto/awslambda/models.py
+++ b/moto/awslambda/models.py
@@ -59,8 +59,6 @@
 
 logger = logging.getLogger(__name__)
 
-docker_3 = docker.__version__[0] >= "3"
-
 
 def zip2tar(zip_bytes: bytes) -> bytes:
     with TemporaryDirectory() as td:
@@ -127,11 +125,7 @@ def __enter__(self) -> "_DockerDataVolumeContext":
             self._vol_ref.volume = self._lambda_func.docker_client.volumes.create(
                 self._lambda_func.code_digest
             )
-            volumes = {
-                self.name: {"bind": "/tmp/data", "mode": "rw"}
-                if docker_3
-                else "/tmp/data"
-            }
+            volumes = {self.name: {"bind": settings.LAMBDA_DATA_DIR, "mode": "rw"}}
 
             self._lambda_func.docker_client.images.pull(
                 ":".join(parse_image_ref("alpine"))
@@ -141,7 +135,7 @@ def __enter__(self) -> "_DockerDataVolumeContext":
             )
             try:
                 tar_bytes = zip2tar(self._lambda_func.code_bytes)
-                container.put_archive("/tmp/data", tar_bytes)
+                container.put_archive(settings.LAMBDA_DATA_DIR, tar_bytes)
             finally:
                 container.remove(force=True)
 
@@ -801,14 +795,11 @@ def _invoke_lambda(self, event: Optional[str] = None) -> Tuple[str, bool, str]:
                 finally:
                     if container:
                         try:
-                            exit_code = container.wait(timeout=300)
+                            exit_code = container.wait(timeout=300)["StatusCode"]
                         except requests.exceptions.ReadTimeout:
                             exit_code = -1
                             container.stop()
                             container.kill()
-                        else:
-                            if docker_3:
-                                exit_code = exit_code["StatusCode"]
 
                         output = container.logs(stdout=False, stderr=True)
                         output += container.logs(stdout=True, stderr=False)
@@ -1514,6 +1505,12 @@ def lambda_handler(event, context):
 
         MOTO_DOCKER_LAMBDA_IMAGE=mlupin/docker-lambda
 
+    Use the following environment variable if you want to configure the data directory used by the Docker containers:
+
+    .. sourcecode:: bash
+
+        MOTO_LAMBDA_DATA_DIR=/tmp/data
+
     .. note:: When using the decorators, a Docker container cannot reach Moto, as the Docker-container loses all mock-context. Any boto3-invocations used within your Lambda will try to connect to AWS.
     """
 
diff --git a/moto/settings.py b/moto/settings.py
--- a/moto/settings.py
+++ b/moto/settings.py
@@ -35,6 +35,8 @@
 # Fully skip test that require docker
 SKIP_REQUIRES_DOCKER = bool(os.environ.get("TESTS_SKIP_REQUIRES_DOCKER", False))
 
+LAMBDA_DATA_DIR = os.environ.get("MOTO_LAMBDA_DATA_DIR", "/tmp/data")
+
 
 def get_sf_execution_history_type():
     """


Test patch: