Repository: docker/compose

Issue:
[BUG] docker compose cp doesn't work for services executed via docker compose run
### Description

The very useful `docker compose cp` command does not work when I use `docker compose run {service}`.

### Steps To Reproduce

Example `docker-compose.yml`:

```
services:
  foo:
    image: alpine
```

What works:

```
docker compose up
docker compose cp foo:/etc/alpine-release ./
docker compose down
```

What doesn't work:

```
docker compose run foo echo
docker compose cp foo:/etc/alpine-release ./
```

Error:

```
no container found for service "foo"
```

Expectation:
It works the same with `docker compose up` or at least there is another way to copy the file.

### Compose Version

```Text
Docker Compose version v2.15.1 (both, docker compose version and docker-compose version)
```


### Docker Environment

```Text
docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc., v0.10.3)
  dev: Docker Dev Environments (Docker Inc., v0.1.0)
  extension: Manages Docker extensions (Docker Inc., v0.2.18)
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc., 0.6.0)
  scan: Docker Scan (Docker Inc., v0.25.0)
  scout: Command line tool for Docker Scout (Docker Inc., v0.6.0)

Server:
 Containers: 4
  Running: 0
  Paused: 0
  Stopped: 4
 Images: 54
 Server Version: 20.10.23
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2 io.containerd.runtime.v1.linux
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 2456e983eb9e37e47538f59ea18f2043c9a73640
 runc version: v1.1.4-0-g5fd4c4d
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
  cgroupns
 Kernel Version: 5.15.49-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 5
 Total Memory: 11.68GiB
 Name: docker-desktop
 ID: YCB4:AIWC:GZ2L:IWPO:MGNR:4JOP:AUQC:KNJQ:KPOR:CMD6:GUIJ:ETLS
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: true
 Insecure Registries:
  hubproxy.docker.internal:5000
  127.0.0.0/8
 Live Restore Enabled: false
```


### Anything else?

_No response_

Hints from issue comments:
@kamilpi: Hi @synek317 

I would like to say "It's not a bug but a feature" and if you will explore the approach and differences between ```run``` and ```up``` you can see what is going on inside.

In short words:

The ```run``` command collects your service definition but in the end, it is just a ```docker``` run command prepared by your service definition.

I give you some insight into this case. Here is a part of the code when ```docker compose``` service definition is collecting https://github.com/docker/compose/blob/v2/cmd/compose/run.go#L219 at the very end do the docker run the command without attaching to the service: https://github.com/docker/compose/blob/v2/pkg/compose/run.go#L41. This is one-off container created.

This one-off container option has one rule. As you may probably know in the runtime process docker creates additional labels to be able to manage services. Based on this I will just show you different labels:

```run```:
```
"com.docker.compose.oneoff": "True"
```

```up```:
```
"com.docker.compose.oneoff": "False"
```

Compose uses also this label to list services: https://github.com/docker/compose/blob/v2/pkg/compose/ps.go#L32

Will preempt your next question: You are not able to override this label.


In the end, I have a solution for you, only the command, without explanation. Check this out:

```
docker-compose run --name foo foo
docker cp foo:/etc/alpine-release ./
```
@ndeloof: Could you please explain what use-case would make this feature useful ?

As a workaround, you obviously can use `docker cp ..` for one-off containers (container started by `docker compose run ...`).@ndeloof: agree _ in theory_, but I noticed we let you `docker compose exec` into a one off container if no service is running for target service, so would like to better understand the use-case for this feature request@synek317: @ndeloof 

My use case is test development for CLI application that downloads files from various online services (webservices, ftps etc.). I have a docker-compose.yml with multiple services. One of them (the one that I run) is my CLI app, others are the aforementioned external services.

`docker compose run` has many advantages: it ends once my app ends, it allows me to pass arguments to my application. Initially, I just used bind volume but I quickly encountered an issue with file permissions on Linux. Basically, files created by the docker container on the host machine were owned by root:root what breaks further processing.  

This is how I discovered `docker compose cp`. It doesn't have this issue but it doesn't work with `docker compose run`. So I had to develop another workaround, which is `docker compose up --abort-on-container-exit --exit-code-from`. Which then needed yet another workaround to pass cmd line arguments to my service (I use env variable for that).

I think that you can see how many times did I use the word `workaround`. It's definitely not perfect.

And in the end, it is counter-intuitive that `docker compose cp` works with _some_ containers created with `docker compose` but not with _others_. It becomes clearer when you know the underlying details that @kamilpi explained. Still, I think being intuitive even for newcomers makes tools better.@iamthad: I, too, was confused that I can `docker compose exec` in containers created by `docker compose run` but I cannot `docker compose cp` to them.
@ndeloof: the main reason we have this limitation is that `docker compose cp` targets _multiple_ containers (all replicas for service), while `docker compose exec` only target one@memory-efficient-irrepla: @ndeloof https://aws-amplify-cli-puppy.github.io/5YFvVAcQek/@ndeloof: @memory-efficient-irrepla AFAICT this question is about Dockerfile `COPY`, unrelated to this issue

Solution patch:
diff --git a/cmd/compose/cp.go b/cmd/compose/cp.go
index 9cd07e5f..bd6281fc 100644
--- a/cmd/compose/cp.go
+++ b/cmd/compose/cp.go
@@ -66,9 +66,7 @@ func copyCommand(p *ProjectOptions, dockerCli command.Cli, backend api.Service)
 
 	flags := copyCmd.Flags()
 	flags.IntVar(&opts.index, "index", 0, "Index of the container if service has multiple replicas")
-	flags.BoolVar(&opts.all, "all", false, "Copy to all the containers of the service")
-	flags.MarkHidden("all")                                                                                                     //nolint:errcheck
-	flags.MarkDeprecated("all", "By default all the containers of the service will get the source file/directory to be copied") //nolint:errcheck
+	flags.BoolVar(&opts.all, "all", false, "Include containers created by the run command")
 	flags.BoolVarP(&opts.followLink, "follow-link", "L", false, "Always follow symbol link in SRC_PATH")
 	flags.BoolVarP(&opts.copyUIDGID, "archive", "a", false, "Archive mode (copy all uid/gid information)")
 
diff --git a/docs/reference/compose_cp.md b/docs/reference/compose_cp.md
index 9ed14140..0886bbd9 100644
--- a/docs/reference/compose_cp.md
+++ b/docs/reference/compose_cp.md
@@ -7,6 +7,7 @@ Copy files/folders between a service container and the local filesystem
 
 | Name                  | Type   | Default | Description                                             |
 |:----------------------|:-------|:--------|:--------------------------------------------------------|
+| `--all`               | `bool` |         | Include containers created by the run command           |
 | `-a`, `--archive`     | `bool` |         | Archive mode (copy all uid/gid information)             |
 | `--dry-run`           | `bool` |         | Execute command in dry run mode                         |
 | `-L`, `--follow-link` | `bool` |         | Always follow symbol link in SRC_PATH                   |
diff --git a/docs/reference/docker_compose_cp.yaml b/docs/reference/docker_compose_cp.yaml
index 8ff3cf37..24f6aec8 100644
--- a/docs/reference/docker_compose_cp.yaml
+++ b/docs/reference/docker_compose_cp.yaml
@@ -10,9 +10,9 @@ options:
     - option: all
       value_type: bool
       default_value: "false"
-      description: Copy to all the containers of the service
-      deprecated: true
-      hidden: true
+      description: Include containers created by the run command
+      deprecated: false
+      hidden: false
       experimental: false
       experimentalcli: false
       kubernetes: false
diff --git a/go.mod b/go.mod
index 4a14e175..8cce9a6a 100644
--- a/go.mod
+++ b/go.mod
@@ -7,7 +7,7 @@ require (
 	github.com/Microsoft/go-winio v0.6.2
 	github.com/acarl005/stripansi v0.0.0-20180116102854-5a71ef0e047d
 	github.com/buger/goterm v1.0.4
-	github.com/compose-spec/compose-go/v2 v2.3.1-0.20241015085011-35c9659361ef
+	github.com/compose-spec/compose-go/v2 v2.4.0
 	github.com/containerd/containerd v1.7.22
 	github.com/containerd/platforms v0.2.1
 	github.com/davecgh/go-spew v1.1.1
diff --git a/go.sum b/go.sum
index f70a5462..5f0bd8a8 100644
--- a/go.sum
+++ b/go.sum
@@ -85,8 +85,8 @@ github.com/cncf/xds/go v0.0.0-20231128003011-0fa0005c9caa h1:jQCWAUqqlij9Pgj2i/P
 github.com/cncf/xds/go v0.0.0-20231128003011-0fa0005c9caa/go.mod h1:x/1Gn8zydmfq8dk6e9PdstVsDgu9RuyIIJqAaF//0IM=
 github.com/codahale/rfc6979 v0.0.0-20141003034818-6a90f24967eb h1:EDmT6Q9Zs+SbUoc7Ik9EfrFqcylYqgPZ9ANSbTAntnE=
 github.com/codahale/rfc6979 v0.0.0-20141003034818-6a90f24967eb/go.mod h1:ZjrT6AXHbDs86ZSdt/osfBi5qfexBrKUdONk989Wnk4=
-github.com/compose-spec/compose-go/v2 v2.3.1-0.20241015085011-35c9659361ef h1:E3qLbOpEyqemgAkQQg3uKNFaJJ+cVXGKiy1Xj4zh49k=
-github.com/compose-spec/compose-go/v2 v2.3.1-0.20241015085011-35c9659361ef/go.mod h1:lFN0DrMxIncJGYAXTfWuajfwj5haBJqrBkarHcnjJKc=
+github.com/compose-spec/compose-go/v2 v2.4.0 h1:Hb0YNtknMWZGHUuULtV/f0cYJVTLt7TWHGEtJwOPzK4=
+github.com/compose-spec/compose-go/v2 v2.4.0/go.mod h1:lFN0DrMxIncJGYAXTfWuajfwj5haBJqrBkarHcnjJKc=
 github.com/containerd/cgroups v1.1.0 h1:v8rEWFl6EoqHB+swVNjVoCJE8o3jX7e8nqBGPLaDFBM=
 github.com/containerd/cgroups v1.1.0/go.mod h1:6ppBcbh/NOOUU+dMKrykgaBnK9lCIBxHqJDGwsa1mIw=
 github.com/containerd/console v1.0.4 h1:F2g4+oChYvBTsASRTz8NP6iIAi97J3TtSAsLbIFn4ro=
diff --git a/pkg/compose/cp.go b/pkg/compose/cp.go
index 107af78d..f65e7919 100644
--- a/pkg/compose/cp.go
+++ b/pkg/compose/cp.go
@@ -61,11 +61,6 @@ func (s *composeService) copy(ctx context.Context, projectName string, options a
 		direction |= fromService
 		serviceName = srcService
 		copyFunc = s.copyFromContainer
-
-		// copying from multiple containers of a services doesn't make sense.
-		if options.All {
-			return errors.New("cannot use the --all flag when copying from a service")
-		}
 	}
 	if destService != "" {
 		direction |= toService
@@ -80,7 +75,7 @@ func (s *composeService) copy(ctx context.Context, projectName string, options a
 		return errors.New("unknown copy direction")
 	}
 
-	containers, err := s.listContainersTargetedForCopy(ctx, projectName, options.Index, direction, serviceName)
+	containers, err := s.listContainersTargetedForCopy(ctx, projectName, options, direction, serviceName)
 	if err != nil {
 		return err
 	}
@@ -119,18 +114,22 @@ func (s *composeService) copy(ctx context.Context, projectName string, options a
 	return g.Wait()
 }
 
-func (s *composeService) listContainersTargetedForCopy(ctx context.Context, projectName string, index int, direction copyDirection, serviceName string) (Containers, error) {
+func (s *composeService) listContainersTargetedForCopy(ctx context.Context, projectName string, options api.CopyOptions, direction copyDirection, serviceName string) (Containers, error) {
 	var containers Containers
 	var err error
 	switch {
-	case index > 0:
-		ctr, err := s.getSpecifiedContainer(ctx, projectName, oneOffExclude, true, serviceName, index)
+	case options.Index > 0:
+		ctr, err := s.getSpecifiedContainer(ctx, projectName, oneOffExclude, true, serviceName, options.Index)
 		if err != nil {
 			return nil, err
 		}
 		return append(containers, ctr), nil
 	default:
-		containers, err = s.getContainers(ctx, projectName, oneOffExclude, true, serviceName)
+		withOneOff := oneOffExclude
+		if options.All {
+			withOneOff = oneOffInclude
+		}
+		containers, err = s.getContainers(ctx, projectName, withOneOff, true, serviceName)
 		if err != nil {
 			return nil, err
 		}
diff --git a/pkg/compose/create.go b/pkg/compose/create.go
index 6994823a..639c1fba 100644
--- a/pkg/compose/create.go
+++ b/pkg/compose/create.go
@@ -977,10 +977,6 @@ func buildContainerConfigMounts(p types.Project, s types.ServiceConfig) ([]mount
 			target = configsBaseDir + config.Target
 		}
 
-		if config.UID != "" || config.GID != "" || config.Mode != nil {
-			logrus.Warn("config `uid`, `gid` and `mode` are not supported, they will be ignored")
-		}
-
 		definedConfig := p.Configs[config.Source]
 		if definedConfig.External {
 			return nil, fmt.Errorf("unsupported external config %s", definedConfig.Name)
@@ -997,6 +993,10 @@ func buildContainerConfigMounts(p types.Project, s types.ServiceConfig) ([]mount
 			continue
 		}
 
+		if config.UID != "" || config.GID != "" || config.Mode != nil {
+			logrus.Warn("config `uid`, `gid` and `mode` are not supported, they will be ignored")
+		}
+
 		bindMount, err := buildMount(p, types.ServiceVolumeConfig{
 			Type:     types.VolumeTypeBind,
 			Source:   definedConfig.File,
@@ -1027,10 +1027,6 @@ func buildContainerSecretMounts(p types.Project, s types.ServiceConfig) ([]mount
 			target = secretsDir + secret.Target
 		}
 
-		if secret.UID != "" || secret.GID != "" || secret.Mode != nil {
-			logrus.Warn("secrets `uid`, `gid` and `mode` are not supported, they will be ignored")
-		}
-
 		definedSecret := p.Secrets[secret.Source]
 		if definedSecret.External {
 			return nil, fmt.Errorf("unsupported external secret %s", definedSecret.Name)
@@ -1047,6 +1043,10 @@ func buildContainerSecretMounts(p types.Project, s types.ServiceConfig) ([]mount
 			continue
 		}
 
+		if secret.UID != "" || secret.GID != "" || secret.Mode != nil {
+			logrus.Warn("secrets `uid`, `gid` and `mode` are not supported, they will be ignored")
+		}
+
 		if _, err := os.Stat(definedSecret.File); os.IsNotExist(err) {
 			logrus.Warnf("secret file %s does not exist", definedSecret.Name)
 		}
@@ -1363,7 +1363,6 @@ func (s *composeService) resolveExternalNetwork(ctx context.Context, n *types.Ne
 
 	switch len(networks) {
 	case 1:
-		n.Name = networks[0].ID
 		return nil
 	case 0:
 		enabled, err := s.isSWarmEnabled(ctx)



Test patch:
diff --git a/pkg/e2e/watch_test.go b/pkg/e2e/watch_test.go
index 4f7ebda2..d7b56435 100644
--- a/pkg/e2e/watch_test.go
+++ b/pkg/e2e/watch_test.go
@@ -99,9 +99,8 @@ func TestRebuildOnDotEnvWithExternalNetwork(t *testing.T) {
 				errors)
 	}, 30*time.Second, 1*time.Second)
 
-	n := c.RunDockerCmd(t, "network", "inspect", networkName, "-f", "{{ .Id }}")
 	pn := c.RunDockerCmd(t, "inspect", containerName, "-f", "{{ .HostConfig.NetworkMode }}")
-	assert.Equal(t, pn.Stdout(), n.Stdout())
+	assert.Equal(t, strings.TrimSpace(pn.Stdout()), networkName)
 
 	t.Log("create a dotenv file that will be used to trigger the rebuild")
 	err = os.WriteFile(dotEnvFilepath, []byte("HELLO=WORLD\nTEST=REBUILD"), 0o666)
@@ -119,9 +118,8 @@ func TestRebuildOnDotEnvWithExternalNetwork(t *testing.T) {
 		return true, fmt.Sprintf("container %s was rebuilt", containerName)
 	}, 30*time.Second, 1*time.Second)
 
-	n2 := c.RunDockerCmd(t, "network", "inspect", networkName, "-f", "{{ .Id }}")
 	pn2 := c.RunDockerCmd(t, "inspect", containerName, "-f", "{{ .HostConfig.NetworkMode }}")
-	assert.Equal(t, pn2.Stdout(), n2.Stdout())
+	assert.Equal(t, strings.TrimSpace(pn2.Stdout()), networkName)
 
 	assert.Check(t, !strings.Contains(r.Combined(), "Application failed to start after update"))