Issue 1:
Docker does a credential check when pulling images, but not when using them. It will let you run an image that is already pulled even if you don't present credentials.

That is reasonable behavior from the standpoint of the docker tool since it is not trying to be multi-user.

However, it makes it hard to make Kubernetes multi-user. If an image is already pulled, (e.g. using imagePullSecrets, then anyone else can use that image by setting pull policy to IfNotPresent, even if they lack proper credentials.

We could try to fix this by disallowing IfNotPresent pull policy on all pods, thus forcing a docker pull before each docker run, and failing the pod if the docker pull failed. We would need to implement and e2e test that.

Issue 2:
Problem statement

Some images require pull secrets in order to be used by the Kubelet. Pull secrets are per namespace. Imagine the following scenario:

If a pod in namespaceA has valid pull secrets for foo/bar:stable, when it is scheduled to a given node, the Kubelet uses the pull secrets to pull the image to the node

If a pod in namespaceB also refers to foo/bar:stable but does not have valid pull secrets for it, if it happens to be scheduled to the same node as the pod in step 1, the Kubelet respects the image pull policy

Because the image already exists on the node, if the pull policy is Never or IfNotPresent, the Kubelet runs the pod, even though namespaceB shouldn't be able to use that image.

Proposed solutions

Option 1 - Kubelet flag to ignore pull policy and always try to pull

Add a flag to the kubelet, --always-pull-images (or something similarly named) that, if true, would force the Kubelet to attempt to pull every image prior to running a pod, regardless of the image's pull policy.

Pros:

Easy to implement

Guarantees that a namespace is allowed to use an image

Cons:

Overrides user-specified intent in image pull policy

Requires remote registry to be online to check authorization

Introduces small amount of latency to pod startup time (because of the additional network requests to the registry)

Option 2 - Keep an in-memory cache of which namespaces are allowed to use which images

After an image has been successfully pulled, record the mapping from namespace to image in an in-memory cache in the Kubelet. Subsequent attempts by pods in the same namespace to use the same image would respect the image pull policy. Any attempts by pods in other namespaces to use this image would ignore the image pull policy and instead force a pull attempt.

Pros:

Doesn't override user-specified intent in image pull policy

No added latency to pod startup time

Cons:

Doesn't handle authorization revocation; if the credentials to pull an image have changed in the remote registry, there is no way to evict an entry from the cache, possibly allowing access to an image that would otherwise fail with option 1. This could be mitigated if the entries in the cache have a TTL.

Suggested path forward

Looking for feedback from the community on this one. I'd like to implement option 1 in the short term, with the default being false (so image pull policy is respected) to keep the status quo. If we can come to an agreement on option 2, I'd like to move to that eventually.

@kubernetes/goog-node @kubernetes/rh-cluster-infra @kubernetes/sig-node