Repository: garden-io/garden

Issue #1831: cleanup-cluster-registry tries to remove images in use
Created: 2020-05-13T16:23:28Z
Bug
Current Behavior

The command garden --env=dev plugins kubernetes cleanup-cluster-registry fails with the following error:
Cleaning up caches and unused images from the in-cluster registry ⚙️

ℹ providers                 → Getting status...
✔ providers                 → Getting status... → Done

ℹ Scanning all Pods in the cluster...
   Found 66 images in use in cluster, 38 referencing the in-cluster registry.
✔ Scanning all Pods in the cluster...
ℹ Listing all images in cluster registry...
   Found 19 images in the registry.
✔ Listing all images in cluster registry...
ℹ Flagging 0 unused images as deleted in cluster registry...
   Flagged 0 images as deleted in the registry.
✔ Flagging 0 unused images as deleted in cluster registry...
ℹ Running garbage collection in cluster registry...
   Fetching original Deployment
   Re-starting in read-only mode...
   ℹ docker-registry           → Waiting for resources to be ready...
   ℹ docker-registry           → Resources ready
   Running garbage collection...
   Restarting without read-only mode...
   ℹ docker-registry           → Waiting for resources to be ready...
   ℹ docker-registry           → Resources ready
   Completed registry garbage collection.
✔ Running garbage collection in cluster registry...
ℹ Cleaning images from Docker daemon...
   Getting list of images from daemon...
   28 tagged images in daemon.
   Cleaning up 1 batches of images (total of 1 images)...

Command "/Users/emanuele/.garden/tools/kubectl/b57d7033ebdf2a5a/kubectl --context=gke --kubeconfig=XXXX --namespace=garden-system exec -i garden-docker-daemon-546769bf5b-nsx9x -c docker-daemon -- docker rmi node:12-alpine" failed with code 1:

Error response from daemon: conflict: unable to remove repository reference "node:12-alpine" (must force) - container 806bbebada2b is using its referenced image 4b3c4c2702d2
command terminated with exit code 1
and it looks like it's because it's trying to remove images that are in use (node:12-alpine)?
Expected behavior

Reproducible example


Workaround

A (non-ideal) workaround for this is to uninstall the Garden system services from the cluster, and then reinstall them:
garden plugins kubernetes uninstall-garden-services && garden plugins kubernetes cluster-init

Your environment


garden version latest master df9e4e47c93cfc227f1e31af33e15c135e64e409
kubectl version
docker version

Duplicate issue #1789: k8s: cleanup-cluster-registry fails when it's unable to remove a repository reference
Created: 2020-04-09T11:01:15Z
Bug
Sometimes, garden plugins kubernetes cleanup-cluster-registry will fail because the Docker daemon is unable to remove a repository reference.
Note: This is when using in-cluster building with a remote cluster.
Current Behavior
$ garden plugins kubernetes cleanup-cluster-registry 
Cleaning up caches and unused images from the in-cluster registry :gear: 
ℹ providers                 → Getting status...
ℹ providers                 → Preparing environment...
   ℹ kubernetes                → Configuring...
   ⚠ One or more cluster-wide system services are outdated or their configuration does not match your current configuration. You may want to run garden --env=dev plugins kubernetes cluster-init to update them, or contact a cluster admin to do so.
   ✔ kubernetes                → Configuring... → Ready
✔ providers                 → Preparing environment... → Done
ℹ Scanning all Pods in the cluster...
   Found 52 images in use in cluster, 25 referencing the in-cluster registry.
✔ Scanning all Pods in the cluster...
ℹ Listing all images in cluster registry...
   Found 19 images in the registry.
✔ Listing all images in cluster registry...
ℹ Flagging 0 unused images as deleted in cluster registry...
   Flagged 0 images as deleted in the registry.
✔ Flagging 0 unused images as deleted in cluster registry...
ℹ Running garbage collection in cluster registry...
   Fetching original Deployment
   Re-starting in read-only mode...
   ℹ docker-registry           → Waiting for resources to be ready...
   ⚠ docker-registry           → Deployment/garden-docker-registry: Multi-Attach error for volume "pvc-d252c8bd-5297-46a3-aa88-183854d128d0" Volume is already exclusively attached to one node and can't be attached to another
   ⚠ docker-registry           → Deployment/garden-docker-registry: AttachVolume.Attach succeeded for volume "pvc-d252c8bd-5297-46a3-aa88-183854d128d0" 
   ⚠ docker-registry           → Deployment/garden-docker-registry: Started container docker-registry
   ℹ docker-registry           → Resources ready
   Running garbage collection...
   Restarting without read-only mode...
   ℹ docker-registry           → Waiting for resources to be ready...
   ℹ docker-registry           → Resources ready
   Completed registry garbage collection.
✔ Running garbage collection in cluster registry...
ℹ Cleaning images from Docker daemon...
   Getting list of images from daemon...
   25 tagged images in daemon.
   Cleaning up 1 batches of images (total of 1 images)...
Command "/Users/foo/.garden/tools/kubectl/b57d7033ebdf2a5a/kubectl --context=gke --kubeconfig=/Users/foo/projects/myproject/services/infra/gke-cluster-dev-f29377c0.yaml --namespace=garden-system exec -i garden-docker-daemon-6dcd85dc-7kkhq -c docker-daemon -- docker rmi node:12-alpine" failed with code 1:
Error response from daemon: conflict: unable to remove repository reference "node:12-alpine" (must force) - container b9c0d74e04c2 is using its referenced image f77abbe89ac1
command terminated with exit code 1
See error.log for detailed error message
 Cleaning images from Docker daemon...

Expected behavior
garden plugins kubernetes cleanup-cluster-registry should run to completion.
Reproducible example
None comes to mind.
Workaround
A workaround for this is to uninstall the Garden system services from the cluster, and then reinstall them:
garden plugins kubernetes uninstall-garden-services && garden plugins kubernetes cluster-init

Suggested solution(s)

Additional context

Your environment

garden version -> 0.11.11
kubectl version
Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.5", GitCommit:"20c265fef0741dd71a66480e35bd69f18351daea", GitTreeState:"clean", BuildDate:"2019-10-15T19:16:51Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"15+", GitVersion:"v1.15.11-gke.3", GitCommit:"71631703d37fb1b4b333abd427deae839aeb2032", GitTreeState:"clean", BuildDate:"2020-03-24T18:54:48Z", GoVersion:"go1.12.17b4", Compiler:"gc", Platform:"linux/amd64"}

docker version
N/A (was using a remote cluster without Docker running locally).

Closing comment (2020-05-29T11:06:03Z): Closing as duplicate of #1831