Repository: PrefectHQ/prefect

Issue:
Error while deploying flow: FileNotFoundError(2, "No such file or directory: 'ip'")
## Description

Using code snippet from https://docs.prefect.io/orchestration/tutorial/docker.html#persisting-your-flow-with-docker-storage 

```python
import prefect
from prefect import task, Flow
from prefect.environments.storage import Docker

@task
def hello_task():
    logger = prefect.context.get("logger")
    logger.info("Hello, Docker!")

flow = Flow("hello-docker", tasks=[hello_task])

flow.storage = Docker()

flow.register()
```

running it in the container ends up building proper image and registering the flow in local server:

```
...
 ---> 3d851b191d56
Successfully built 3d851b191d56
Successfully tagged hello-docker:2020-04-20t08-03-01-759322-00-00
Flow: http://localhost:8080/flow/5af6b16e-cbd1-44f0-b855-80e3420f4860
```

Then I start the agent `prefect agent start docker` (inside the same container or different one, which joined the same docker network and has docker unix socket access via mounted volume).

The error I observe:

```
root@876bfe616714:~/flows# prefect agent start docker

 ____            __           _        _                    _
|  _ \ _ __ ___ / _| ___  ___| |_     / \   __ _  ___ _ __ | |_
| |_) | '__/ _ \ |_ / _ \/ __| __|   / _ \ / _` |/ _ \ '_ \| __|
|  __/| | |  __/  _|  __/ (__| |_   / ___ \ (_| |  __/ | | | |_
|_|   |_|  \___|_|  \___|\___|\__| /_/   \_\__, |\___|_| |_|\__|
                                           |___/

[2020-04-20 08:09:06,297] INFO - agent | Starting DockerAgent with labels []
[2020-04-20 08:09:06,298] INFO - agent | Agent documentation can be found at https://docs.prefect.io/cloud/
[2020-04-20 08:09:06,299] INFO - agent | Waiting for flow runs...
[2020-04-20 08:09:42,514] INFO - agent | Found 1 flow run(s) to submit for execution.
[2020-04-20 08:09:42,589] INFO - agent | Deploying flow run b86a2022-4e17-49cb-b5d2-266c6adf3593
[2020-04-20 08:09:42,601] ERROR - agent | Logging platform error for flow run b86a2022-4e17-49cb-b5d2-266c6adf3593
[2020-04-20 08:09:42,702] ERROR - agent | Error while deploying flow: FileNotFoundError(2, "No such file or directory: 'ip'")
```


## Expected Behavior

Pick the flow and execute it.


## Reproduction

1. Start prefect stack locally (docker compose)
2. Start additional container and join the compose network, e.g. `docker run -it --rm -v $(pwd)/flows:/root/flows -v /var/run/docker.sock:/var/run/docker.sock --network "prefect-server" prefecthq/prefect:latest /bin/bash`
3. Init the env:
```
export PREFECT__BACKEND=server
export PREFECT__SERVER__HOST=http://docker.for.mac.host.internal
```
4. Register the flow (see above)
5. Run agent (see above)


## Environment

OSX, Docker Desktop Community 2.2.0.5



Discussion:
Thanks for opening @oleksandr! Since you're running the Docker agent inside a container my first thought is that it doesn't know where the daemon is in order to connect. You could try specifying the location of your Docker daemon through the agent's argument:

```python
- base_url (str, optional): URL for a Docker daemon server. Defaults to
            `unix:///var/run/docker.sock` however other hosts such as
            `tcp://0.0.0.0:2375` can be provided
```

Will need to do some more digging if that does not solve it. I have seen this `"No such file or directory: 'ip'"` issue before and IIRC it was due to someone attempting to run a container inside a container.
@joshmeek I've did further digging and traced this error to function call `get_docker_ip()`, which appears to call `ip` binary from `iproute2` package (at least on ubuntu). This command and corresponding package is not available in the official Prefect docker image.

I've installed it manually, but obviously it doesn't work as expected - there's no `docker0` inteface inside of a container: https://github.com/PrefectHQ/prefect/blob/master/src/prefect/utilities/docker_util.py#L13

Ignoring this issue and trying still to run agent in DinD fashion (mounting docker socket via volume or using `socat` and exposing it via TCP in docker network) leads to a new issue:

1. I build flow with Docker storage (image is available)
2. Run docker agent (inside of container)
3. Run the flow
4. Observe how agent picks the task and fails at the creation of a container with an error: 

```
400 Client Error: Bad Request ("invalid IP address in add-host: """)
```
I'm looking into this issue now, but overall using current implementation of `get_docker_ip()` is limiting use-cases.

Ah I see what you are saying about `get_docker_ip`. Since the container base is linux it is always calling that even though the function is more for users running the Docker agent on their Linux machines and not container. For that we should allow an opt-out of that check.

The `add-host` error looks like there may be an issue with the exposed daemon endpoint. Any more context on the setup you did prior?
@joshmeek I will try to reset the environment today, reproduce the issue and share the details here to be able to move forward.
So here're details on the environment I'm running. This is local setup, but it reflects our production setup, which is Dockerized, but without an orchestrator (independent machines). This article shades some light: https://www.redhat.com/sysadmin/podman-shareable-systemd-services - pretty much each container is managed by systemd.

My local setup:

1. The latest Prefect stack is started as usually on OSX, no changes to that. 

2. The flow is basic with the following relevant parts (Docker for storage, S3 for results, Dask/Processes as env for parallelization withing the same container):

```python

    flow = ...

    flow.environment = RemoteEnvironment(
        executor="prefect.engine.executors.LocalDaskExecutor",
        executor_kwargs={"scheduler": "processes", "num_workers": 12},
    )

    flow.storage = Docker(image_name='hello-docker', image_tag='latest')
    flow.storage.build()

    flow.result_handler = S3ResultHandler(
        "flows",
        boto3_kwargs=dict(endpoint_url=os.getenv("AWS_S3_ENDPOINT"))
    )

```

3. I do task registration from within a container that joins prefect-server network, pass host system's docker socket via volume (`-v /var/run/docker.sock:/var/run/docker.sock`) and that works fine:
```bash
...
 ---> Running in 81497fa9350e
Beginning health checks...
System Version check: OK
Cloudpickle serialization check: OK
Result Handler check: OK
Environment dependency check: OK
All health checks passed.
Removing intermediate container 81497fa9350e
 ---> 39765d0ad452
Successfully built 39765d0ad452
Successfully tagged hello-docker:latest
Flow: http://localhost:8080/flow/b1c9dd6a-0716-4b1b-8f8f-ee14c8379cf1
```

4. I start separate container that also joins prefect-server network, also pass host system's docker socket (`-v /var/run/docker.sock:/var/run/docker.sock `) with the following entrypoint:
```
prefect agent start docker --base-url=unix:///var/run/docker.sock --network=prefect-server -l docker
```

5. I trigger the flow from UI and observe the following in agent's log:
```
[2020-04-24 13:20:22,909] INFO - agent | Found 1 flow run(s) to submit for execution.
[2020-04-24 13:20:22,993] INFO - agent | Deploying flow run a69fb0fe-41fb-4d4a-b91b-1530025d3d57
[2020-04-24 13:20:23,058] ERROR - agent | Logging platform error for flow run a69fb0fe-41fb-4d4a-b91b-1530025d3d57
[2020-04-24 13:20:23,129] ERROR - agent | Error while deploying flow: APIError(HTTPError('400 Client Error: Bad Request for url: http+docker://localhost/v1.40/containers/create'))
```
I also tried to switch to TCP socket for docker. 

1. Adding such service to docker network exposes socket as `tcp://docker:2375`:

```yaml
  docker:
    image: alpine/socat:latest
    networks:
    - prefect-server
    ports:
      - "2375:2375"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    command: tcp-listen:2375,fork,reuseaddr unix-connect:/var/run/docker.sock
```

2. I passed it to agent via `base_url` and observed the same error as above.
I wonder if this has something to do with the Docker client's `networking_config`. We do something like this:

```python
networking_config = None
if self.network:
    networking_config = self.docker_client.create_networking_config(
        {self.network: self.docker_client.create_endpoint_config()}
    )
```

Bad request (400) leads me to think something we're sending isn't correct or compatible and it seems that the only thing "out of the ordinary" would be the network. When you tried the switch to TCP did you also provide a network of `prefect-server`?
@joshmeek The lines above are fine, network is reachable (I do pass network name in both cases).

I think the error is still related to `get_docker_ip`. I've tried to comment out these lines:

```python
        if sys.platform.startswith("linux"):
            docker_internal_ip = get_docker_ip()
            host_config.update(extra_hosts={"host.docker.internal": docker_internal_ip})
```

and dockerized flow has been executed. Looks like in DinD cases specifying `extra_hosts` with empty value causes this 400 error (according to docs it should always be `hostname:ip`, but `docker_internal_ip` is empty because `docker0` interface doesn't exist inside of a container.
@oleksandr Okay gotcha, I'm going to add a flag that you can set to disable that check 👍 

Resolving patch:
diff --git a/src/prefect/agent/docker/agent.py b/src/prefect/agent/docker/agent.py
--- a/src/prefect/agent/docker/agent.py
+++ b/src/prefect/agent/docker/agent.py
@@ -50,6 +50,9 @@ class DockerAgent(Agent):
             to stdout; defaults to `False`
         - volumes (List[str], optional): a list of Docker volume mounts to be attached to any and all created containers.
         - network (str, optional): Add containers to an existing docker network
+        - docker_interface (bool, optional): Toggle whether or not a `docker0` interface is present on this machine.
+            Defaults to `True`. **Note**: This is mostly relevant for some Docker-in-Docker setups that users may be
+            running their agent with.
     """
 
     def __init__(
@@ -63,6 +66,7 @@ def __init__(
         volumes: List[str] = None,
         show_flow_logs: bool = False,
         network: str = None,
+        docker_interface: bool = True,
     ) -> None:
         super().__init__(
             name=name, labels=labels, env_vars=env_vars, max_polls=max_polls
@@ -94,6 +98,11 @@ def __init__(
         self.network = network
         self.logger.debug("Docker network set to {}".format(self.network))
 
+        self.docker_interface = docker_interface
+        self.logger.debug(
+            "Docker interface toggle set to {}".format(self.docker_interface)
+        )
+
         self.failed_connections = 0
         self.docker_client = self._get_docker_client()
         self.show_flow_logs = show_flow_logs
@@ -339,7 +348,7 @@ def deploy_flow(self, flow_run: GraphQLResult) -> str:
         if container_mount_paths:
             host_config.update(binds=self.host_spec)
 
-        if sys.platform.startswith("linux"):
+        if sys.platform.startswith("linux") and self.docker_interface:
             docker_internal_ip = get_docker_ip()
             host_config.update(extra_hosts={"host.docker.internal": docker_internal_ip})
 
diff --git a/src/prefect/cli/agent.py b/src/prefect/cli/agent.py
--- a/src/prefect/cli/agent.py
+++ b/src/prefect/cli/agent.py
@@ -119,6 +119,12 @@ def agent():
 @click.option(
     "--network", help="Add containers to an existing docker network", hidden=True,
 )
+@click.option(
+    "--no-docker-interface",
+    is_flag=True,
+    help="Disable presence of a Docker interface.",
+    hidden=True,
+)
 @click.pass_context
 def start(
     ctx,
@@ -136,6 +142,7 @@ def start(
     show_flow_logs,
     volume,
     network,
+    no_docker_interface,
     max_polls,
 ):
     """
@@ -171,12 +178,15 @@ def start(
 
     \b
     Docker Agent Options:
-        --base-url, -b  TEXT    A Docker daemon host URL for a DockerAgent
-        --no-pull               Pull images for a DockerAgent
-                                Defaults to pulling if not provided
-        --volume        TEXT    Host paths for Docker bind mount volumes attached to each Flow runtime container.
-                                Multiple values supported e.g. `--volume /some/path --volume /some/other/path`
-        --network       TEXT    Add containers to an existing docker network
+        --base-url, -b      TEXT    A Docker daemon host URL for a DockerAgent
+        --no-pull                   Pull images for a DockerAgent
+                                    Defaults to pulling if not provided
+        --volume            TEXT    Host paths for Docker bind mount volumes attached to each Flow runtime container.
+                                    Multiple values supported e.g. `--volume /some/path --volume /some/other/path`
+        --network           TEXT    Add containers to an existing docker network
+        --no-docker-interface       Disable the check of a Docker interface on this machine.
+                                    **Note**: This is mostly relevant for some Docker-in-Docker setups that users may be
+                                    running their agent with.
 
     \b
     Kubernetes Agent Options:
@@ -234,6 +244,7 @@ def start(
                 show_flow_logs=show_flow_logs,
                 volumes=list(volume),
                 network=network,
+                docker_interface=not no_docker_interface,
             ).start()
         elif agent_option == "fargate":
             from_qualified_name(retrieved_agent)(


Test patch:
diff --git a/tests/agent/test_docker_agent.py b/tests/agent/test_docker_agent.py
--- a/tests/agent/test_docker_agent.py
+++ b/tests/agent/test_docker_agent.py
@@ -849,3 +849,73 @@ def test_docker_agent_network(monkeypatch, runner_token):
     assert agent.network == "test-network"
     args, kwargs = api.create_container.call_args
     assert kwargs["networking_config"] == {"test-network": "config"}
+
+
+def test_docker_agent_deploy_with_interface_check_linux(
+    monkeypatch, runner_token, linux_platform
+):
+
+    api = MagicMock()
+    api.ping.return_value = True
+    api.create_container.return_value = {"Id": "container_id"}
+    monkeypatch.setattr(
+        "prefect.agent.docker.agent.DockerAgent._get_docker_client",
+        MagicMock(return_value=api),
+    )
+
+    get_ip = MagicMock()
+    monkeypatch.setattr("prefect.agent.docker.agent.get_docker_ip", get_ip)
+
+    agent = DockerAgent()
+    agent.deploy_flow(
+        flow_run=GraphQLResult(
+            {
+                "flow": GraphQLResult(
+                    {
+                        "storage": Docker(
+                            registry_url="", image_name="name", image_tag="tag"
+                        ).serialize()
+                    }
+                ),
+                "id": "id",
+                "name": "name",
+            }
+        )
+    )
+
+    assert get_ip.called
+
+
+def test_docker_agent_deploy_with_no_interface_check_linux(
+    monkeypatch, runner_token, linux_platform
+):
+
+    api = MagicMock()
+    api.ping.return_value = True
+    api.create_container.return_value = {"Id": "container_id"}
+    monkeypatch.setattr(
+        "prefect.agent.docker.agent.DockerAgent._get_docker_client",
+        MagicMock(return_value=api),
+    )
+
+    get_ip = MagicMock()
+    monkeypatch.setattr("prefect.agent.docker.agent.get_docker_ip", get_ip)
+
+    agent = DockerAgent(docker_interface=False)
+    agent.deploy_flow(
+        flow_run=GraphQLResult(
+            {
+                "flow": GraphQLResult(
+                    {
+                        "storage": Docker(
+                            registry_url="", image_name="name", image_tag="tag"
+                        ).serialize()
+                    }
+                ),
+                "id": "id",
+                "name": "name",
+            }
+        )
+    )
+
+    assert not get_ip.called
diff --git a/tests/cli/test_agent.py b/tests/cli/test_agent.py
--- a/tests/cli/test_agent.py
+++ b/tests/cli/test_agent.py
@@ -203,6 +203,7 @@ def test_agent_start_with_env_vars(monkeypatch, runner_token):
         show_flow_logs=False,
         volumes=[],
         network=None,
+        docker_interface=True,
     )
 
 
@@ -224,6 +225,7 @@ def test_agent_start_with_max_polls(monkeypatch, runner_token):
         show_flow_logs=False,
         volumes=[],
         network=None,
+        docker_interface=True,
     )