Repository: docker/compose

Issue:
Windows: missing secrets file creates file-named directory
<!--
Welcome to the docker-compose issue tracker! Before creating an issue, please heed the following:

1. This tracker should only be used to report bugs and request features / enhancements to docker-compose
    - For questions and general support, use https://forums.docker.com
    - For documentation issues, use https://github.com/docker/docker.github.io
    - For issues with the `docker stack` commands and the version 3 of the Compose file, use
      https://github.com/docker/cli
2. Use the search function before creating a new issue. Duplicates will be closed and directed to
   the original discussion.
3. When making a bug report, make sure you provide all required information. The easier it is for
   maintainers to reproduce, the faster it'll be fixed.
-->

## Description of the issue

On Windows, when starting a composition that refers to a secret file which does not exist, an "invalid mount config for type" error is generated and a directory with the same name as the file is created.

This behavior is *not* seen on Mac, with the same version of Docker Desktop and Docker Compose.

This may be related to #5377 but seems only to affect Windows.

## Context information (for bug reports)

**Output of `docker-compose version`**
```
docker-compose version 1.29.0, build 07737305
docker-py version: 5.0.0
CPython version: 3.9.0
OpenSSL version: OpenSSL 1.1.1g  21 Apr 2020
```

**Output of `docker version`**
```
Client: Docker Engine - Community
 Cloud integration: 1.0.12
 Version:           20.10.5
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        55c4c88
 Built:             Tue Mar  2 20:14:53 2021
 OS/Arch:           windows/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.5
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       363e9a8
  Built:            Tue Mar  2 20:15:47 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.4
  GitCommit:        05f951a3781f4f2c1911b05e61c160e9c30eaa8e
 runc:
  Version:          1.0.0-rc93
  GitCommit:        12644e614e25b05da6fd08a38ffa0cfe1903fdec
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
```

**Output of `docker-compose config`**
(Make sure to add the relevant `-f` and other flags)
```
D:\Repos\Scratch\compose-test>docker-compose config
secrets:
  postgres_db:
    file: D:\Repos\Scratch\compose-test\postgres_db.txt
  postgres_password:
    file: D:\Repos\Scratch\compose-test\postgres_password.txt
  postgres_user:
    file: D:\Repos\Scratch\compose-test\postgres_user.txt
services:
  db:
    environment:
      POSTGRES_DB_FILE: /run/secrets/postgres_db
      POSTGRES_INITDB_ARGS: --auth=md5
      POSTGRES_PASSWORD_FILE: /run/secrets/postgres_password
      POSTGRES_USER_FILE: /run/secrets/postgres_user
    healthcheck:
      interval: 10s
      retries: 5
      test:
      - CMD-SHELL
      - pg_isready -U postgres
      timeout: 5s
    image: postgres:latest
    labels:
      com.microsoft.vscode.dev-container-name: webapi-nodejs-dev-container
    restart: unless-stopped
    secrets:
    - source: postgres_db
    - source: postgres_password
    - source: postgres_user
    volumes:
    - postgres-data:/var/lib/postgresql/data:rw
version: '3.7'
volumes:
  postgres-data: {}
```


## Steps to reproduce the issue

1. Add the following to `docker-compose.yml` in an otherwise empty directory:

```yaml
version: '3.7'

secrets:
  postgres_db:
    file: postgres_db.txt
  postgres_password:
    file: postgres_password.txt
  postgres_user:
    file: postgres_user.txt

services:
  db:
    image: postgres:latest
    restart: unless-stopped
    volumes:
      - postgres-data:/var/lib/postgresql/data
    environment:
      - POSTGRES_DB_FILE=/run/secrets/postgres_db
      - POSTGRES_INITDB_ARGS=--auth=md5
      - POSTGRES_PASSWORD_FILE=/run/secrets/postgres_password
      - POSTGRES_USER_FILE=/run/secrets/postgres_user
    labels:
      com.microsoft.vscode.dev-container-name: webapi-nodejs-dev-container
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 10s
      timeout: 5s
      retries: 5
    secrets:
      - postgres_db
      - postgres_password
      - postgres_user

    # Add "forwardPorts": ["5432"] to **devcontainer.json** to forward Postgress locally.
    # (Adding the "ports" property to this file will not forward from a Codespace.)

volumes:
  postgres-data:
```

2. `docker-compose up --remove-orphans`

### Observed result

1. Composition fails to start (which is expected).
2. Folders are created in the directory with the same names as the declared secrets file (not expected).

> NOTE: This does not repro on every attempt.  What I observe is that it works on the first attempt, but not (always) subsequent attempts.  However, if you tweak the names of the secrets files, then you can (usually) repro it on the next attempt.

### Expected result

1. Composition fails to start.
2. No directories (or files) created.

### Stacktrace / full error message

```
D:\Repos\Scratch\compose-test>docker-compose up --remove-orphans
WARNING: Service "db" uses an undefined secret file "D:\Repos\Scratch\compose-test\postgres_db.txt", the following file should be created "D:\Repos\Scratch\compose-test\postgres_db.txt"
WARNING: Service "db" uses an undefined secret file "D:\Repos\Scratch\compose-test\postgres_password.txt", the 
following file should be created "D:\Repos\Scratch\compose-test\postgres_password.txt"
WARNING: Service "db" uses an undefined secret file "D:\Repos\Scratch\compose-test\postgres_user.txt", the following file should be created "D:\Repos\Scratch\compose-test\postgres_user.txt"
Docker Compose is now in the Docker CLI, try `docker compose up`

Removing orphan container "compose-test_app_1"
Starting compose-test_db_1 ... error

ERROR: for compose-test_db_1  Cannot start service db: invalid mount config for type "bind": bind source path does not exist: /run/desktop/mnt/host/d/Repos/Scratch/compose-test/postgres_db.txt

ERROR: for db  Cannot start service db: invalid mount config for type "bind": bind source path does not exist: 
/run/desktop/mnt/host/d/Repos/Scratch/compose-test/postgres_db.txt
ERROR: Encountered errors while bringing up the project.
```

## Additional information

OS version / distribution, `docker-compose` install method, etc.

Windows 10 (Version 21H1, OS Build 19043.964)

Discussion:
@pmorch: I'm not 100% sure, but I think this issue has gotten worse recently on Docker Deskop for Windows.

If the `file: ./mypass.txt` is missing, it creates a directory in the current directory called `mypass.txt`. It has been like that "for a long time".

(I'm pretty sure) it used to be that I could then:

```
docker-compose down
rmdir mypass.txt
echo contents > mypass.txt
docker-compose up -d
```

and that would be the end of it.

But now when I do that last step, I get:

```
WSL2@~/idea/app» docker-compose up -d
Creating network "app_default" with the default driver
Creating app_app_1 ... error

ERROR: for app_app_1  Cannot create container for service app: not a directory

ERROR: for app  Cannot create container for service app: not a directory
ERROR: Encountered errors while bringing up the project.
```

To get rid of that error, I now have to:

```
WSL2@~/idea/app» sudo rm -r /mnt/wsl/docker-desktop-bind-mounts/Ubuntu-20.04-v2/e47c2d04a4b94e64079cbd50e60e132e8e5670dbd7d9fc84721344f3f2b4e105
```

and then it works:

```
WSL2@~/idea/app» docker-compose up -d
Creating network "app_default" with the default driver
Creating app_app_1 ... done
```

I have no idea how to identify the correct `e47c2d04a4b94e64079cbd50e60e132e8e5670dbd7d9fc84721344f3f2b4e105` file under `/mnt/wsl/docker-desktop-bind-mounts/Ubuntu-20.04-v2` but the timestamp may help and I found that quitting Docker Desktop for Windows and then:

```
WSL2@~/idea/app» sudo rm -r /mnt/wsl/docker-desktop-bind-mounts/Ubuntu-20.04-v2/*
```

And then restarting Docker Desktop works too. 

**Note**: I was worried that this `sudo rm -r .../*`  might loose data somehow, but I didn't have any data I didn't mind loosing. You might. Be careful and investigate what you're deleting... Even though I didn't mind loosing them, I checked and all my volume contents were still intact afterwards

```
WSL2@~/idea/app» docker-compose version
docker-compose version 1.29.1, build c34c88b2
docker-py version: 5.0.0
CPython version: 3.7.10
OpenSSL version: OpenSSL 1.1.0l  10 Sep 2019
```
@spwoodcock: For anyone reading the comment above - be VERY careful following the advice to delete some directories under /mnt/wsl/docker-desktop-bind-mounts/

It has the potential to not only delete Docker volumes, but **delete your entire WSL filesystem**. I just found out the hard way.

I'm not sure if this is to do with a recent update to Docker Desktop and some sort of symlinking, but I really hope this secrets mounting issue can be fixed at some point.@delyada: It's not just about missing secrets file. **My secrets files are there but I get the error.**

I'm in a WSL 2 VS Code dev container.

- If I use [docker-from-docker](https://github.com/microsoft/vscode-dev-containers/blob/main/script-library/docs/docker.md) (i.e. the Windows host's Docker) to up my composition, the error happens.
- If I use [docker-in-docker](https://github.com/microsoft/vscode-dev-containers/blob/main/script-library/docs/docker-in-docker.md) (i.e. the Linux container's internal installation of Docker) to up my composition, no such error happens.@JustEnoughDucks: Interestingly, this happens to me on my brand new debian baremetal docker install. 

On my old debian server which has docker at the exact same version of every component, but was upgraded starting from an early 2020 docker version does not have this issue. The difference was the parent folder permissions.

The difference between them for me was that the secrets directory (not the files themselves) needs group _execute_ permissions. I don't know how applicable it is for windows, but it seems to be a permissions issue?@stale: This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
@MicahZoltu: Still a problem, current behavior:
```yml
version: '3.8'

secrets:
  my_secret:
    file: my_secret.txt

services:
  hello-world:
    image: hello-world
    secrets:
      - my_secret
```
```bash
> docker-compose up
WARNING: Service "hello-world" uses an undefined secret file "C:\Users\micah\Source\temp\my_secret.txt", the following file should be created "C:\Users\micah\Source\temp\my_secret.txt"
...
```
* `my_secret.txt` directory is created
* The compose executes, despite the missing secret file (it should fail).
* If you delete the folder and run `docker-compose up` again, you now get a failure:
  	```bash
	  > docker-compose up
	  WARNING: Service "hello-world" uses an undefined secret file "C:\...\my_secret.txt", the following file should be created "C:\...\my_secret.txt"
	  Starting temp_hello-world_1 ... error
	  
	  ERROR: for temp_hello-world_1  Cannot start service hello-world: invalid mount config for type "bind": bind source path does not exist: /run/desktop/mnt/host/c/.../my_secret.txt
	  ```
* Deleting the folder and then creating a text file in its place results in the following failure:
  	```bash
	  > docker-compose up
	  Starting temp_hello-world_1 ... error
	  
	  ERROR: for temp_hello-world_1  Cannot start service hello-world: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: rootfs_linux.go:76: mounting "/run/desktop/mnt/host/c/.../my_secret.txt" to rootfs at "/run/secrets/my_secret" caused: mount through procfd: not a directory: unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type
	  ```
* Restarting docker reverts behavior back to initial behavior.
* Behavior is the same when run from a WSL terminal with docker client installed (connected up to Docker for Windows).@MicahZoltu: @stale Not stale.@MicahZoltu: The above replicated on:
Docker Desktop: 4.4.4 (73704
Docker Engine: 20.10.12
Docker Compose: 1.29.2@glours: FYI Docker Compose V1, the python version, reached end-of-life, please use Compose v2 instead and let us know if you still have the problem with this version.
You can simply switch to Compose V2 by checking the checkbox in Docker Desktop preferences
<img width="1125" alt="Screenshot 2022-07-31 at 11 34 50" src="https://user-images.githubusercontent.com/705411/182020234-289f8279-34a7-490e-b04f-448825c8b006.png">@MicahZoltu: Tested with Docker Compose v2 per above instructions:
New behavior (still broken):
* I get no warning at all, and a folder is created (as before).
* If I delete the folder after the first run then `docker-compose up` again I get the following error:
  ```bash
  > docker-compose up
  [+] Running 1/0
   - Container temp-hello-world-1  Created
  Attaching to temp-hello-world-1
  Error response from daemon: invalid mount config for type "bind": bind source path does not exist: /run/desktop/mnt/host/c/.../my_secret.txt
  ```
* Restarting Docker no longer reverts behavior back to initial behavior, it now seems to get permanently stuck on the second-run behavior.
* Creating the secret after second `docker-compose up` attempt results in this error:
  ```
  > docker-compose up
  [+] Running 1/0
   - Container temp-hello-world-1  Created
  Attaching to temp-hello-world-1
  Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: rootfs_linux.go:76: mounting "/run/desktop/mnt/host/c/.../my_secret.txt" to rootfs at "/run/secrets/my_secret" caused: mount through procfd: not a directory: unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type
  ```@stale: This issue has been automatically marked as not stale anymore due to the recent activity.
@stale: This issue has been automatically marked as not stale anymore due to the recent activity.
@digitalkelvin: Any progress? I have same issue as @MicahZoltu @dollusion: I also have this issue and get the same result now with v3@LordLumineer: It appears to be fixed
@Progress1: It looks not :-)  (Latest Docker Desktop 4.30.0, latest updated Windows)
- if secrets are missing first time, it creates directory with secret names
- then i delete manually directories and create secrets files
- after using docker-compose up i get error:
```
Error response from daemon: failed to create task for container: failed to create shim task: 
OCI runtime create failed: runc create failed: unable to start container process: error during container init: 
error mounting "/run/desktop/mnt/host/c/_/Dev/XXX/docker/secrets/postgres_password.txt" to rootfs at "/run/secrets/postgres_password": 
mount /run/desktop/mnt/host/c/_/Dev/XXX/docker/secrets/postgres_password.txt:/run/secrets/postgres_password (via /proc/self/fd/6), 
flags: 0x5000: not a directory: unknown: 
Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type

```
file `c:\_\Dev\XXX\docker\secrets\postgres_password.txt`  exists.

docker-compose.yml:
```
services:
  database:
    ...
    secrets:
       - postgres_password
secrets:
  postgres_password:
    file: ./secrets/postgres_password.txt

```    
Any solution to get secrets working?

**UPDATE/FIX:** i comment secrets definition, recreate container (this probably delete old mounts), un-comment secrets, recreate container again (now are mounts correctly mapped) and it start without error@dakkafex: The only way i have gotten it to work sofar is by running `docker-compose up --build` by going into the WSL terminal first.@ndeloof: > if secrets are missing first time, it creates directory with secret names

this indeed is the default behavior for a bind mount. Proposed an additional check to enforce source exists before creating container with bind mount for secret@Kylotan: I'm still seeing this problem with Docker Compose version v2.32.4-desktop.1 on Windows.

Solution patch:
diff --git a/pkg/compose/convergence.go b/pkg/compose/convergence.go
index 5ad3c94b..4f90cda9 100644
--- a/pkg/compose/convergence.go
+++ b/pkg/compose/convergence.go
@@ -364,7 +364,12 @@ func containerReasonEvents(containers Containers, eventFunc func(string, string)
 const ServiceConditionRunningOrHealthy = "running_or_healthy"
 
 //nolint:gocyclo
-func (s *composeService) waitDependencies(ctx context.Context, project *types.Project, dependant string, dependencies types.DependsOnConfig, containers Containers) error {
+func (s *composeService) waitDependencies(ctx context.Context, project *types.Project, dependant string, dependencies types.DependsOnConfig, containers Containers, timeout time.Duration) error {
+	if timeout > 0 {
+		withTimeout, cancelFunc := context.WithTimeout(ctx, timeout)
+		defer cancelFunc()
+		ctx = withTimeout
+	}
 	eg, _ := errgroup.WithContext(ctx)
 	w := progress.ContextWriter(ctx)
 	for dep, config := range dependencies {
@@ -454,7 +459,11 @@ func (s *composeService) waitDependencies(ctx context.Context, project *types.Pr
 			}
 		})
 	}
-	return eg.Wait()
+	err := eg.Wait()
+	if errors.Is(err, context.DeadlineExceeded) {
+		return fmt.Errorf("timeout waiting for dependencies")
+	}
+	return err
 }
 
 func shouldWaitForDependency(serviceName string, dependencyConfig types.ServiceDependency, project *types.Project) (bool, error) {
@@ -760,12 +769,12 @@ func (s *composeService) isServiceCompleted(ctx context.Context, containers Cont
 	return false, 0, nil
 }
 
-func (s *composeService) startService(ctx context.Context, project *types.Project, service types.ServiceConfig, containers Containers) error {
+func (s *composeService) startService(ctx context.Context, project *types.Project, service types.ServiceConfig, containers Containers, timeout time.Duration) error {
 	if service.Deploy != nil && service.Deploy.Replicas != nil && *service.Deploy.Replicas == 0 {
 		return nil
 	}
 
-	err := s.waitDependencies(ctx, project, service.Name, service.DependsOn, containers)
+	err := s.waitDependencies(ctx, project, service.Name, service.DependsOn, containers, timeout)
 	if err != nil {
 		return err
 	}
diff --git a/pkg/compose/create.go b/pkg/compose/create.go
index c4d1734f..f4178d6a 100644
--- a/pkg/compose/create.go
+++ b/pkg/compose/create.go
@@ -1003,11 +1003,18 @@ func buildContainerSecretMounts(p types.Project, s types.ServiceConfig) ([]mount
 			continue
 		}
 
+		if _, err := os.Stat(definedSecret.File); os.IsNotExist(err) {
+			logrus.Warnf("secret file %s does not exist", definedSecret.Name)
+		}
+
 		mnt, err := buildMount(p, types.ServiceVolumeConfig{
 			Type:     types.VolumeTypeBind,
 			Source:   definedSecret.File,
 			Target:   target,
 			ReadOnly: true,
+			Bind: &types.ServiceVolumeBind{
+				CreateHostPath: false,
+			},
 		})
 		if err != nil {
 			return nil, err
diff --git a/pkg/compose/run.go b/pkg/compose/run.go
index cdafb4b8..1bd769db 100644
--- a/pkg/compose/run.go
+++ b/pkg/compose/run.go
@@ -93,7 +93,7 @@ func (s *composeService) prepareRun(ctx context.Context, project *types.Project,
 	}
 
 	if !opts.NoDeps {
-		if err := s.waitDependencies(ctx, project, service.Name, service.DependsOn, observedState); err != nil {
+		if err := s.waitDependencies(ctx, project, service.Name, service.DependsOn, observedState, 0); err != nil {
 			return "", err
 		}
 	}
diff --git a/pkg/compose/start.go b/pkg/compose/start.go
index 7caeb305..b5fcceeb 100644
--- a/pkg/compose/start.go
+++ b/pkg/compose/start.go
@@ -129,7 +129,7 @@ func (s *composeService) start(ctx context.Context, projectName string, options
 			return err
 		}
 
-		return s.startService(ctx, project, service, containers)
+		return s.startService(ctx, project, service, containers, options.WaitTimeout)
 	})
 	if err != nil {
 		return err
@@ -149,7 +149,7 @@ func (s *composeService) start(ctx context.Context, projectName string, options
 			defer cancel()
 		}
 
-		err = s.waitDependencies(ctx, project, project.Name, depends, containers)
+		err = s.waitDependencies(ctx, project, project.Name, depends, containers, 0)
 		if err != nil {
 			if errors.Is(ctx.Err(), context.DeadlineExceeded) {
 				return fmt.Errorf("application not healthy after %s", options.WaitTimeout)



Test patch:
diff --git a/pkg/compose/convergence_test.go b/pkg/compose/convergence_test.go
index e25ccd9f..dc981fda 100644
--- a/pkg/compose/convergence_test.go
+++ b/pkg/compose/convergence_test.go
@@ -239,7 +239,7 @@ func TestWaitDependencies(t *testing.T) {
 			"db":    {Condition: ServiceConditionRunningOrHealthy},
 			"redis": {Condition: ServiceConditionRunningOrHealthy},
 		}
-		assert.NilError(t, tested.waitDependencies(context.Background(), &project, "", dependencies, nil))
+		assert.NilError(t, tested.waitDependencies(context.Background(), &project, "", dependencies, nil, 0))
 	})
 	t.Run("should skip dependencies with condition service_started", func(t *testing.T) {
 		dbService := types.ServiceConfig{Name: "db", Scale: intPtr(1)}
@@ -252,7 +252,7 @@ func TestWaitDependencies(t *testing.T) {
 			"db":    {Condition: types.ServiceConditionStarted, Required: true},
 			"redis": {Condition: types.ServiceConditionStarted, Required: true},
 		}
-		assert.NilError(t, tested.waitDependencies(context.Background(), &project, "", dependencies, nil))
+		assert.NilError(t, tested.waitDependencies(context.Background(), &project, "", dependencies, nil, 0))
 	})
 }