Change:
/kind bug We're having trouble retrieving images from a private registry on DockerCloud. For nearly a year the following has worked flawlessly. Last Friday I pulled down the 1.9.0 CLI (via brew) and since then have not been able to access any image in a private registry on DockerCloud. I've tried this on 10+ brand new clusters created with kops v1.8.0. The Kubernetes cluster version have been 1.8.0, 1.8.4, 1.8.5 and 1.9.0 (which is not supported by kops yet) and all exhibit the same problem. We create the secret as such: And verify it using: Which yields: And create a test service with with these contents: The pod fails immediatly with: I've reviewed the 1.9.0 release notes and nothing jumps out as the root cause. For reference I've tried putting quotes around the value when creating the secret to no avail. I've also tried prefixing the value with , so the full value would read: also to no avail. Environment: Kubernetes version: Cloud provider and installation info: AWS cluster built with Kops 1.8.0, using (to isolate this issue) nearly the default settings for everything.
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I have same problem, and I "solved" it creating a docker-registry secret in an old kubernetes version and pasting it on new kubernetes
I'm recently (I know it worked perfectly about last month) facing issues with as well. For me it ended in a similar manner saying: I was able to confirm, that this is a kubernetes CLI related issue and not end user. Based on and this I was able to create properly formatted auth information looking like this: . When encoding this one-liner using (MacOS workstation) and apply like this works fine and images can be pulled from private repository again. However this breaks all automation and can't be considered as solution Confirmed working with kubectl 1.8.5 and server 1.8.4 (guess 1.7.9 as well)
looks like there's an extra "auths" element when adding the secret using 1.9 kubectl
Added in /assign
Many thanks for the quick fix on this. Any idea when this might get rolled out? Also, it might be a good idea if this was to the known issue list for v1.9.0. It would have saved us a bunch of time trying to track down the root cause. Just trying to make sure no one else needs to do the same.
linking to the "v1.9.0 known issues" list
ack, asking Mehdy since he is the patch manager for v1.9 branch. Mehdy, any idea when we can ship 1.9.1 with this fix?
workarounds for folks until 1.9.1 is released: if you have a .dockerconfigjson for your private registry already, you can manually specify the type and data key (e.g. ) if you don't have a .dockerconfigjson file already, you can fix up the secret produced by manually: the type from to the data key from to the modified secret with
1 will be released tomorrow
I can confirm solution works, thanks so much!
I am still having issues in 1.9.1. This worked in the 1.8.x branches: results in the following error:
you have to remove the gitlab-registry secret before you can apply an update... type is immutable, and the fix in 1.9.1 required changing the type of the secret

Discussion:
kubectl create secret docker-registry test-secret --docker-username=test-user --docker-password=test-password --docker-email='test-user@test.com' --namespace=test-secrets # Post-condition: secret exists and has expected values kube::test::get_object_assert 'secret/test-secret --namespace=test-secrets' "{{$id_field}}" 'test-secret' <del> kube::test::get_object_assert 'secret/test-secret --namespace=test-secrets' "{{$secret_type}}" 'kubernetes.io/dockercfg' [[ "$(kubectl get secret/test-secret --namespace=test-secrets -o yaml "${kube_flags[@]}" | grep '.dockercfg:')" ]] </del> <ins> kube::test::get_object_assert 'secret/test-secret --namespace=test-secrets' "{{$secret_type}}" 'kubernetes.io/dockerconfigjson' [[ "$(kubectl get secret/test-secret --namespace=test-secrets -o yaml "${kube_flags[@]}" | grep '.dockerconfigjson:')" ]] </ins> # Clean-up kubectl delete secret test-secret --namespace=test-secrets

if err := s.validate(); err != nil { return nil, err } <del> dockercfgContent, err := handleDockercfgContent(s.Username, s.Password, s.Email, s.Server) </del> <ins> dockercfgJsonContent, err := handleDockerCfgJsonContent(s.Username, s.Password, s.Email, s.Server) </ins> if err != nil { return nil, err } secret := &v1.Secret{} secret.Name = s.Name <del> secret.Type = v1.SecretTypeDockercfg </del> <ins> secret.Type = v1.SecretTypeDockerConfigJson </ins> secret.Data = map[string][]byte{} <del> secret.Data[v1.DockerConfigKey] = dockercfgContent </del> <ins> secret.Data[v1.DockerConfigJsonKey] = dockercfgJsonContent </ins> if s.AppendHash { h, err := hash.SecretHash(secret) if err != nil {

return nil } <del> // handleDockercfgContent serializes a dockercfg json file func handleDockercfgContent(username, password, email, server string) ([]byte, error) { </del> <ins> // handleDockerCfgJsonContent serializes a ~/.docker/config.json file func handleDockerCfgJsonContent(username, password, email, server string) ([]byte, error) { </ins> dockercfgAuth := credentialprovider.DockerConfigEntry{ Username: username, Password: password, Email:    email, } <del> dockerCfg := credentialprovider.DockerConfigJson{ </del> <ins> dockerCfgJson := credentialprovider.DockerConfigJson{ </ins> Auths: map[string]credentialprovider.DockerConfigEntry{server: dockercfgAuth}, } <del> return json.Marshal(dockerCfg) </del> <ins> return json.Marshal(dockerCfgJson) </ins> }

func TestSecretForDockerRegistryGenerate(t *testing.T) { username, password, email, server := "test-user", "test-password", "test-user@example.org", "https://index.docker.io/v1/" <del> secretData, err := handleDockercfgContent(username, password, email, server) </del> <ins> secretData, err := handleDockerCfgJsonContent(username, password, email, server) </ins> if err != nil { t.Errorf("unexpected error: %v", err) } <del> secretDataNoEmail, err := handleDockercfgContent(username, password, "", server) </del> <ins> secretDataNoEmail, err := handleDockerCfgJsonContent(username, password, "", server) </ins> if err != nil { t.Errorf("unexpected error: %v", err) }

Name: "foo", }, Data: map[string][]byte{ <del> v1.DockerConfigKey: secretData, </del> <ins> v1.DockerConfigJsonKey: secretData, </ins> }, <del> Type: v1.SecretTypeDockercfg, </del> <ins> Type: v1.SecretTypeDockerConfigJson, </ins> }, expectErr: false, },

}, expected: &v1.Secret{ ObjectMeta: metav1.ObjectMeta{ <del> Name: "foo-94759gc65b", </del> <ins> Name: "foo-548cm7fgdh", </ins> }, Data: map[string][]byte{ <del> v1.DockerConfigKey: secretData, </del> <ins> v1.DockerConfigJsonKey: secretData, </ins> }, <del> Type: v1.SecretTypeDockercfg, </del> <ins> Type: v1.SecretTypeDockerConfigJson, </ins> }, expectErr: false, },

Name: "foo", }, Data: map[string][]byte{ <del> v1.DockerConfigKey: secretDataNoEmail, </del> <ins> v1.DockerConfigJsonKey: secretDataNoEmail, </ins> }, <del> Type: v1.SecretTypeDockercfg, </del> <ins> Type: v1.SecretTypeDockerConfigJson, </ins> }, expectErr: false, },