Code change:
bumped the vendored docker library and introduced a regression where pods with images that don't contain an explicit "" repo path result in failures. Results in the following error: Pod gets shows that the image has been pulled, though not by that name Changing the image to in the pod spec corrects the issue. I think the new code that is changing the behavior is somewhere in here is a new function brought in by the bump. If I checkout the commit before the bump, the problem does not occur. related:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig node
taking a look now
What docker version are you using?
Do you have any e2e tests that are failing? We run a lot of tests with image "busybox"
Didn't think about it before, but I guess 1.12 is the supported version for 1.8 eh? Let me try it on 1.12.
Hm, weird. We also have cluster e2e test coverage for 1.13. Many test are using directly from this function
I installed a completely fresh box with docker 1.12 and could still recreate. I have no explanation :confused: from kubelet built from upstream/release-1.8
This is one of our e2e test node log: You could find tons of . One thing I don't understand on your side is that, logs in our test are always: Why on your node it's ? Where does that image come from? What if you delete it?
does this mean that upgrades will fail if the local image names need to change?
Seems not. Based on the following code, it's fine to pull/inspect with new reference when we previously pulled with old reference. Could not reproduce this.
I could not reproduce with the code above either.
For any image which has no tag, kubelet will add a default tag , and default repo to it. So will change to and send to dockerd to pull. But dockerd knows and are the same thing, so should return right instead of not found. (as demo) Can you make sure your dockerd is the a version compatible with k8s 1.8?
Just ran into this. If you up the kubelet's log level to 4, you'll see a "not found" error from For example: If we look at the code in that helper function, we have this: Here is the result of inspecting the image on my system (only showing relevant fields): You'll notice that the RepoTag it has is for , not . I am guessing the fact that we're somewhere inserting "library" into the image's name is what's tripping this up.
If it wasn't clear, I believe dockershim is what's having the issue. Once the kubelet sends the protobuf message to the dockershim socket to pull the image, it goes from to to to and then into the helper function I listed above.
It appears to be this: to to to That's where the "library" is coming from.
Prior to the PR that bumped docker/distribution, used to have , which does not do any normalization (i.e. including "library").
OK, I see why when we bumped to the newer docker/distribution, we switch to using . It's because now requires that the input be in "canonical" format (in this case, has and , if applicable), which you get automatically by calling ...
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/node: Issue will be escalated to these SIGs if needed. priority/critical-urgent: Never automatically move out of a release milestone; continually escalate to contributor and SIG through all available channels. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details
/assign
could you give steps to reproduce this? We can't reproduce this and many of the node/cluster e2e tests pull images using unqualified name such as "buxybox"
What docker distribution are you using? I could not reproduce this myself. I don't understand how does the reference show up, we never pull image with it. Is this a problem specific to a docker distribution?
run the e2e tests on a centos/fedora/rhel image
Is there any signal in test-grid for centos/fedora/rhel? Would be nice to catch issues like this sooner than later.
run the e2e tests on a centos/fedora/rhel image there is not, and definitely should be, given that k8s releases rpms as part of each release. will be on the short list for the release retrospective.
I feel like this is a problem in docker distribution in centos/fedora/rhel. Docker should not return when we are using to pull image. Should we fix the corresponding docker distribution instead of blocking Kubernete release?
Since that would mean releasing a new version would regress existing users, we generally say that we'll do our best to tolerate it instead of releasing something known broken. We've done that with various versions of Docker, systemd, and iptables for years, and I'm sure we'll do our best to do that. I do agree that this looks like a bug in that docker distribution that should be fixed. I also agree that this should have been in the test grid and that the distro owner is responsible for ensuring the project has that signal.
Thanks! I agree that we should not break existing users. I'm fine with as long as we keep in mind we made this change, and clean it up in the future. I think it would be helpful if we could have test coverage in test-grid on more distros, so that we could have caught similar problems earlier.
Agree - I think anyone who causes a release slip is required to make sure they fix the problem so it can't happen in the future. And get better all around testing. On Sep 27, 2017, at 7:40 PM, Lantao Liu <: ! I agree that we should not break existing users. I'm fine with long as we keep in mind we made this change, and clean it up in the future. I think it would be helpful if we could have test coverage on more distros, so that we could have caught similar problems earlier. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub , or mute the thread
I have opened a bug for this
This has broken the CRI as well. Kubernetes should not normalize image names on behalf of the CRI runtime. It's up to implementors to decide how to handle an image name and since there's not a standard around Kube and Image names, I'd prefer not to break the CRI.
Are you saying it's currently broken, or the fix was ok? , Antonio Murdaca <:
The fix is ok but the original patch caused another bug
Which one?
caused a bug fixed by
Does this problem fixed?

Related discussion:
text: // hostname or not, we only check for the suffix match. if strings.HasSuffix(image, tag) || strings.HasSuffix(tag, image) { return true <ins> } else { // TODO: We need to remove this hack when project atomic based // docker distro(s) like centos/fedora/rhel image fix problems on // their end. // Say the tag is "docker.io/busybox:latest" // and the image is "docker.io/library/busybox:latest" t, err := dockerref.ParseNormalizedNamed(tag) if err != nil { continue } // the parsed/normalized tag will look like // reference.taggedReference { // 	 namedRepository: reference.repository { // 	   domain: "docker.io", // 	   path: "library/busybox" //	}, // 	tag: "latest" // } // If it does not have tags then we bail out t2, ok := t.(dockerref.Tagged) if !ok { continue } // normalized tag would look like "docker.io/library/busybox:latest" // note the library get added in the string normalizedTag := t2.String() if normalizedTag == "" { continue } if strings.HasSuffix(image, normalizedTag) || strings.HasSuffix(normalizedTag, image) { return true } </ins> } } }