Repository: moby/moby (language: go)

Issue:
Docker daemon won't start if a registry-mirror is configured in /etc/docker/daemon.json
**Description**

I have a Nexus 3 docker mirror running in a local server, which my machine points to using /etc/docker/daemon.json. I've tested it a few times, got it working, all good.

However, I was using a ridiculously old version of docker and decided to update it in order to use multistage builds (the version I was using couldn't handle the "AS" alias for stages), in fact, the newest I found(stable).

I completely removed docker, but left the daemon.json there, since I would need it anyway. Installed version 17.12.1 using a ".deb" and it immediately showed an error after the installation was done, showing that docker daemon was unable to start.

When trying to manually start it using dockerd, I got the following message:
>invalid mirror: path, query, or fragment at end of the URI "http://192.168.1.201:8081/repository/docker-public/"

After about 30 minutes of searching, I've decided to try a different version, just to see what happens, and after downloading version 17.09.1, it started working again

**Steps to reproduce the issue:**
1. Create /etc/docker/daemon.json and insert a registry-mirror, like the following: 
```
{
        "registry-mirrors": ["http://192.168.1.201:8081/repository/docker-public/"]
}
```
2. Update docker to version 17.12.1 using this [link](https://download.docker.com/linux/ubuntu/dists/xenial/pool/stable/amd64/docker-ce_17.12.1~ce-0~ubuntu_amd64.deb)

**Describe the results you received:**
Error when starting docker daemon

**Describe the results you expected:**
Docker daemon should've started as usual

**Additional information you deem important (e.g. issue happens only occasionally):**
Also happens with 17.12.0

**Output of `docker version`:**

```
Client:
 Version:	17.12.1-ce
 API version:	1.35
 Go version:	go1.9.4
 Git commit:	7390fc6
 Built:	Tue Feb 27 22:17:40 2018
 OS/Arch:	linux/amd64
```

**Output of `docker info`:**

```
Can't print, since daemon won't start.
```

**Additional environment details (AWS, VirtualBox, physical, etc.):**
Using a regular ubuntu 16.04 installation behind a corporate firewall (usually a problem, but not this time I guess)


Discussion / hints:
As far as I know, registry mirrors never supported a _path_, but older versions may have silently ignored this.

Does it work if you remove `repository/docker-public`?
Since it's a Nexus 3 registry-mirror, and not one that I've configured, that path is actually what points which of the repositories is a "docker-proxy".

Removing it in 17.12.1 resulted in docker ignoring the registry-mirror and trying to fetch an image directly. Same in 17.09.0, however leaving the path there in 17.09.0 makes it work.
I'm seeing the same behavior with our Nexus docker group (private repo + proxy for public images). 

Docker version:
```
Client:
 Version:      18.03.1-ce
 API version:  1.37
 Go version:   go1.9.5
 Git commit:   9ee9f40
 Built:        Thu Apr 26 07:17:20 2018
 OS/Arch:      linux/amd64
```

Our `/etc/docker/daemon.json`:
```json
{
  "insecure-registries":[
    "192.168.1.10"
  ],
  "registry-mirrors": [
    "http://192.168.1.10/nexus/repository/docker-group"
  ]
}
```

FWIW, this only seems to be an issue with Docker. Gems, Apt, YUM, NPM, and Maven can all pull through Nexus just fine.
Any update on this? Not being able to use Nexus for our mirror is a huge pain
This is also an issue for me.

Will Docker be adding an option to provide a prefix path for the registry mirror?
Any update on this issue? i got same problem T^T
Had a quick chat with some people on our slack, related to https://github.com/moby/moby/pull/38991#discussion_r271517220, because at the time, containerd did not have support for these.

Since then, containerd looks to have added support https://github.com/containerd/containerd/blob/adeba792f16bff6d8f68d560566ed50202634bac/remotes/docker/registry.go#L71, however, it has not yet been looked into if that can be used (and BuildKit would have to be updated to user/support it, as it currently doesn't)
Any movement on this? I came here after attempting to follow the instructions at https://docs.docker.com/registry/recipes/mirror/ after successfully configuring my local registry (intended to be a pull through cache). I have my docker registry behind an nginx reverse proxy and can successfully `curl https://my.domain/docker-registry/v2/_catalog`, thus establishing that it works... but if I can't actually configure my client(s) to connect to it then setting it up was just a waste of time. :(
I've found a solution 
`(
cat <<EOF
[Service]
ExecStart=
ExecStart=/usr/bin/dockerd -H fd:// --registry-mirror http://$PULL_THROUGT_DN:$REPO_PORT --insecure-registry $PULL_THROUGT_DN:$REPO_PORT --insecure-registry=$SECURE_REGISTRY_DN --insecure-registry=$SECURE_REGISTRY_DN 
EOF
) | sudo tee /etc/systemd/system/docker.service.d/docker-options.conf`
and
`systemctl daemon-reload && systemctl restart docker`
worked with ubuntu

UPD
Domain name without anu subpath
@FloMko does that work with a url where the mirror is at a sub path (https://mirror.example.com/docker/mirror)? Or just if it's a domain name (https://mirror.example.com)?
> @FloMko does that work with a url where the mirror is at a sub path (https://mirror.example.com/docker/mirror)? Or just if it's a domain name (https://mirror.example.com)?

Only FQDN without any subpath (comment has been updated) 
OK - FYI that this issue is specifically documenting trying to start it with a subpath, it's known to work fine without a subpath.
> I'm seeing the same behavior with our Nexus docker group (private repo + proxy for public images).
> 
> Docker version:
> 
> ```
> Client:
>  Version:      18.03.1-ce
>  API version:  1.37
>  Go version:   go1.9.5
>  Git commit:   9ee9f40
>  Built:        Thu Apr 26 07:17:20 2018
>  OS/Arch:      linux/amd64
> ```
> 
> Our `/etc/docker/daemon.json`:
> 
> ```json
> {
>   "insecure-registries":[
>     "192.168.1.10"
>   ],
>   "registry-mirrors": [
>     "http://192.168.1.10/nexus/repository/docker-group"
>   ]
> }
> ```
> 
> FWIW, this only seems to be an issue with Docker. Gems, Apt, YUM, NPM, and Maven can all pull through Nexus just fine.

For Nexus: follow this post https://mtijhof.wordpress.com/2018/07/23/using-nexus-oss-as-a-proxy-cache-for-docker-images. It allows you to serve the repo at a different port, mitigating the need for a path.
I'm observing similar behavior.
Docker version:
```
docker version
Client: Docker Engine - Community
 Version:           20.10.2
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        2291f61
 Built:             Mon Dec 28 16:17:43 2020
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.2
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       8891c58
  Built:            Mon Dec 28 16:15:19 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.3
  GitCommit:        269548fa27e0089a8b8278fc4fc781d7f65a939b
 runc:
  Version:          1.0.0-rc92
  GitCommit:        ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
```

This config doesn't work:

```
{
  "insecure-registries": ["insecure.server.example.com:4567"],
  "data-root": "/home/docker",
  "max-download-attempts": 65535,
  "registry-mirrors": ["https://artifactory.local.corp.net/artifactory"]
}
```
Docker daemon complains in logs: 
`failed to start daemon: invalid mirror: path, query, or fragment at end of the URI "https://artifactory.local.corp.net/artifactory"`

However, if I move `registry-mirrors` to separate pair of curly braces, it works
```
{
  "insecure-registries": ["insecure.server.example.com:4567"],
  "data-root": "/home/docker",
  "max-download-attempts": 65535
},
{
  "registry-mirrors": ["https://artifactory.local.corp.net/artifactory"]
}
```

@wl2776 by using an extra pair of curly braces, you just make docker-daemon to start. In fact everything in the second pair of curly braces is being ignored by the daemon, so your solution doesn't work.
> @wl2776 by using an extra pair of curly braces, you just make docker-daemon to start. In fact everything in the second pair of curly braces is being ignored by the daemon, so your solution doesn't work.

Interesting. Does it mean that to use our internal artifactory server, I just need `docker login artifactory....` ? No need to change system configs?
> > @wl2776 by using an extra pair of curly braces, you just make docker-daemon to start. In fact everything in the second pair of curly braces is being ignored by the daemon, so your solution doesn't work.
> 
> Interesting. Does it mean that to use our internal artifactory server, I just need `docker login artifactory....` ? No need to change system configs?

Nope, simply it means that this section of you config is ignored

```
,
{
  "registry-mirrors": ["https://artifactory.local.corp.net/artifactory"]
}
```

and mirroring doesn't work as intended



> Nope, simply it means that this section of you config is ignored
> 
> ```
> ,
> {
>   "registry-mirrors": ["https://artifactory.local.corp.net/artifactory"]
> }
> ```
> 
> and mirroring doesn't work as intended

But `docker pull` retrieves images. How does it know where to connect?

> > Nope, simply it means that this section of you config is ignored
> > ```
> > ,
> > {
> >   "registry-mirrors": ["https://artifactory.local.corp.net/artifactory"]
> > }
> > ```
> > 
> > 
> > and mirroring doesn't work as intended
> 
> But `docker pull` retrieves images. How does it know where to connect?

It should retrieve images from Docker HUB. 
To check this, try to pull some image which is not in your mirror's cache and then go to Artifactory and check whether it's present. I doubt you will not find the image here (this is how I actually tested your solution).
> > But `docker pull` retrieves images. How does it know where to connect?
> 
> It should retrieve images from Docker HUB.
> To check this, try to pull some image which is not in your mirror's cache and then go to Artifactory and check whether it's present. I doubt you will not find the image here (this is how I actually tested your solution).

Actually, its more simple. :) Repo address is provided in the command line. I've forgot about it. :)

`docker pull artifactory.local.corp.net/blah-blah-blah/../.../`
Hi. I just wanted to link this issue with the workaround described here https://github.com/goharbor/harbor/issues/8082#issuecomment-725694016
The solution works to me as a charm.
The problem ist the validation function:

https://github.com/moby/moby/blob/8d193d81af9cbbe800475d4bb8c529d67a6d8f14/registry/config.go#L323-L340

`uri.Path != "" && uri.Path != "/"` does not allow subpaths.

Also to see in unit tests:

https://github.com/moby/moby/blob/8d193d81af9cbbe800475d4bb8c529d67a6d8f14/registry/config_test.go#L131-L160
Our solution was to reverse proxy our registry using another vhost pointing to a subpath of nexus. Extra config, but easy enough since nexus is already behind a reverse proxy anyway.
Wild that we all have to set up elaborate reverse-proxy systems to get this to work for private registry mirrors. The last update here from a project maintainer seemed to indicate that this was a limitation years ago... but yet here we are in 2023 still going "I have to do what to make this work?"
This is a hard problem for us (in my company) - Harbor mirror should be set to `/hub` project. Podman is very easy to configure to fix this, but Docker / Moby fails.

Can a Moby developer to react on this issue ?

Fix patch:
diff --git a/registry/config.go b/registry/config.go
index 2766306ac2093..88f0e64edba27 100644
--- a/registry/config.go
+++ b/registry/config.go
@@ -328,8 +328,8 @@ func ValidateMirror(val string) (string, error) {
 	if uri.Scheme != "http" && uri.Scheme != "https" {
 		return "", invalidParamf("invalid mirror: unsupported scheme %q in %q", uri.Scheme, uri)
 	}
-	if (uri.Path != "" && uri.Path != "/") || uri.RawQuery != "" || uri.Fragment != "" {
-		return "", invalidParamf("invalid mirror: path, query, or fragment at end of the URI %q", uri)
+	if uri.RawQuery != "" || uri.Fragment != "" {
+		return "", invalidParamf("invalid mirror: query or fragment at end of the URI %q", uri)
 	}
 	if uri.User != nil {
 		// strip password from output


Test patch:
diff --git a/registry/config_test.go b/registry/config_test.go
index 77f799df965fc..f68cfc61b2467 100644
--- a/registry/config_test.go
+++ b/registry/config_test.go
@@ -142,21 +142,21 @@ func TestValidateMirror(t *testing.T) {
 		"https://127.0.0.1",
 		"http://127.0.0.1:5000",
 		"https://127.0.0.1:5000",
+		"http://mirror-1.example.com/v1/",
+		"https://mirror-1.example.com/v1/",
 	}
 
 	invalid := []string{
 		"!invalid!://%as%",
 		"ftp://mirror-1.example.com",
 		"http://mirror-1.example.com/?q=foo",
-		"http://mirror-1.example.com/v1/",
 		"http://mirror-1.example.com/v1/?q=foo",
 		"http://mirror-1.example.com/v1/?q=foo#frag",
 		"http://mirror-1.example.com?q=foo",
 		"https://mirror-1.example.com#frag",
 		"https://mirror-1.example.com/#frag",
 		"http://foo:bar@mirror-1.example.com/",
-		"https://mirror-1.example.com/v1/",
-		"https://mirror-1.example.com/v1/#",
+		"https://mirror-1.example.com/v1/#frag",
 		"https://mirror-1.example.com?q",
 	}