Repository: kubernetes/minikube
Pull request #6051 (2019-12-11 03:34:48)

adding feature to reload cache 
this pr:
- add a new command `minikube cache reload` which will reload all the images specified in the config file without having to start-stop
- save 3 seconds on each image ID look up. by using `github.com/docker/docker` ID instead of go-containerregistry .
(worth noting go-containerregistry  uses github.com/docker/docker to load the image id.
https://github.com/google/go-containerregistry/issues/627)

- rename funcs for more clarity and unexport funcs

- added an integration test that does the following :
-- manually delete the image inside the node 
-- verify it doesn't exist anymore.
-- minikube cache reload, should bring the image back.

- refactor image related funcs to its own package

k8s-ci-robot: [APPROVALNOTIFIER] This PR is **APPROVED**

This pull-request has been approved by: *<a href="https://github.com/kubernetes/minikube/pull/6051#" title="Author self-approved">medyagh</a>*

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=kubernetes%2Fminikube).

The pull request process is described [here](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process)

<details >
Needs approval from an approver in each of these files:

- ~~[OWNERS](https://github.com/kubernetes/minikube/blob/master/OWNERS)~~ [medyagh]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->
codecov-io: # [Codecov](https://codecov.io/gh/kubernetes/minikube/pull/6051?src=pr&el=h1) Report
> Merging [#6051](https://codecov.io/gh/kubernetes/minikube/pull/6051?src=pr&el=desc) into [master](https://codecov.io/gh/kubernetes/minikube/commit/ff752fd72923de634e00700a6ae629640daa3ed8?src=pr&el=desc) will **decrease** coverage by `0.01%`.
> The diff coverage is `5.71%`.

[![Impacted file tree graph](https://codecov.io/gh/kubernetes/minikube/pull/6051/graphs/tree.svg?width=650&token=7Xgbyt4fNl&height=150&src=pr)](https://codecov.io/gh/kubernetes/minikube/pull/6051?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #6051      +/-   ##
==========================================
- Coverage   36.74%   36.72%   -0.02%     
==========================================
  Files         113      113              
  Lines        8366     8372       +6     
==========================================
+ Hits         3074     3075       +1     
- Misses       4890     4895       +5     
  Partials      402      402
```


| [Impacted Files](https://codecov.io/gh/kubernetes/minikube/pull/6051?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pkg/minikube/machine/cache\_images.go](https://codecov.io/gh/kubernetes/minikube/pull/6051/diff?src=pr&el=tree#diff-cGtnL21pbmlrdWJlL21hY2hpbmUvY2FjaGVfaW1hZ2VzLmdv) | `4.36% <0%> (-0.02%)` | :arrow_down: |
| [cmd/minikube/cmd/start.go](https://codecov.io/gh/kubernetes/minikube/pull/6051/diff?src=pr&el=tree#diff-Y21kL21pbmlrdWJlL2NtZC9zdGFydC5nbw==) | `22.66% <0%> (ø)` | :arrow_up: |
| [pkg/minikube/machine/cache\_binaries.go](https://codecov.io/gh/kubernetes/minikube/pull/6051/diff?src=pr&el=tree#diff-cGtnL21pbmlrdWJlL21hY2hpbmUvY2FjaGVfYmluYXJpZXMuZ28=) | `84.09% <100%> (ø)` | :arrow_up: |
| [cmd/minikube/cmd/cache.go](https://codecov.io/gh/kubernetes/minikube/pull/6051/diff?src=pr&el=tree#diff-Y21kL21pbmlrdWJlL2NtZC9jYWNoZS5nbw==) | `9.52% <12.5%> (+1.41%)` | :arrow_up: |

medyagh: /ok-to-test
minikube-bot: Error: running mkcmp: exit status 1

minikube-bot: All Times minikube: [ 126.280405 124.846419 127.294431]
All Times Minikube (PR 6051): [ 128.844021 126.396246 127.949744]

Average minikube: **126.140418**
Average Minikube (PR 6051): **127.730004**

Averages Time Per Log
```
+----------------------+-----------+--------------------+
|         LOG          | MINIKUBE  | MINIKUBE (PR 6051) |
+----------------------+-----------+--------------------+
| minikube v           |  0.269908 |           0.279556 |
| Creating kvm2        | 48.151882 |          47.184404 |
| Preparing Kubernetes | 51.352011 |          52.179456 |
| Pulling images       |  2.549397 |           2.976070 |
| Launching Kubernetes | 20.874209 |          21.436444 |
| Waiting for cluster  |  2.893670 |           2.908007 |
+----------------------+-----------+--------------------+
```

medyagh: @tstromberg Thanks for the review !  done.
minikube-bot: All Times Minikube (PR 6051): [ 127.441474 126.053851 112.773293]
All Times minikube: [ 129.211921 126.060465 128.020755]

Average minikube: **127.764380**
Average Minikube (PR 6051): **122.089539**

Averages Time Per Log
```
+----------------------+-----------+--------------------+
|         LOG          | MINIKUBE  | MINIKUBE (PR 6051) |
+----------------------+-----------+--------------------+
| minikube v           |  0.277245 |           0.288176 |
| Creating kvm2        | 47.600327 |          47.644426 |
| Preparing Kubernetes | 52.339941 |          48.551894 |
| Pulling images       |  3.138671 |           3.077702 |
| Launching Kubernetes | 21.629621 |          19.747047 |
| Waiting for cluster  |  2.728557 |           2.728061 |
+----------------------+-----------+--------------------+
```

medyagh: @tstromberg I just verified that the cache add and cache reload do not depend on "docker" binary

I deleted and purged my docker on linux and tried the following
```
medya@medya:~/workspace/minikube (cache_reload)$ docker
bash: docker: command not found

$ ./out/minikube start 
🎉  minikube 1.6.1 is available! Download it: https://github.com/kubernetes/minikube/releases/tag/v1.6.1
💡  To disable this notice, run: 'minikube config set WantUpdateNotification false'

🙄  minikube v1.6.0-beta.1 on Debian rodete
✨  Automatically selected the 'kvm2' driver (alternates: [none])
💾  Downloading driver docker-machine-driver-kvm2:
    > docker-machine-driver-kvm2.sha256: 65 B / 65 B [-------] 100.00% ? p/s 0s
    > docker-machine-driver-kvm2: 13.86 MiB / 13.86 MiB  100.00% 12.70 MiB p/s 
💿  Downloading VM boot image ...
    > minikube-v1.6.0-beta.1.iso.sha256: 65 B / 65 B [-------] 100.00% ? p/s 0s
    > minikube-v1.6.0-beta.1.iso: 149.98 MiB / 149.98 MiB  100.00% 28.81 MiB p/
🔥  Creating kvm2 VM (CPUs=2, Memory=2000MB, Disk=20000MB) ...
🐳  Preparing Kubernetes v1.17.0 on Docker '19.03.5' ...
💾  Downloading kubeadm v1.17.0
💾  Downloading kubelet v1.17.0
🚜  Pulling images ...
🚀  Launching Kubernetes ... 
⌛  Waiting for cluster to come online ...
🏄  Done! kubectl is now configured to use "minikube"
⚠️  /usr/local/google/home/medya/go/bin/kubectl is version 1.15.3, and is incompatible with Kubernetes 1.17.0. You will need to update /usr/local/google/home/medya/go/bin/kubectl or use 'minikube kubectl' to connect with this cluster
medya@medya:~/workspace/minikube (cache_reload)$ ./out/minikube status
host: Running
kubelet: Running
apiserver: Running
kubeconfig: Configured
medya@medya:~/workspace/minikube (cache_reload)$ ./out/minikube cache reload --alsologtostderr -v=9
W1218 11:36:02.485868   23078 root.go:241] Error reading config file at /usr/local/google/home/medya/.minikube/config/config.json: open /usr/local/google/home/medya/.minikube/config/config.json: no such file or directory
medya@medya:~/workspace/minikube (cache_reload)$ ./out/minikube cache reload
medya@medya:~/workspace/minikube (cache_reload)$ ./out/minikube cache add alpine
medya@medya:~/workspace/minikube (cache_reload)$ ./out/minikube cache reload
medya@medya:~/workspace/minikube (cache_reload)$ ./out/minikube cache list
alpine
medya@medya:~/workspace/minikube (cache_reload)$ ./out/minikube ssh crictl images
FATA[0002] failed to connect: failed to connect, make sure you are running as root and the runtime has been started: context deadline exceeded 
ssh: Process exited with status 1
medya@medya:~/workspace/minikube (cache_reload)$ ./out/minikube sudo ssh crictl images
Error: unknown command "sudo" for "minikube"
Run 'minikube --help' for usage.
medya@medya:~/workspace/minikube (cache_reload)$ ./out/minikube ssh sudo crictl images
IMAGE                                     TAG                 IMAGE ID            SIZE
alpine                                    latest              965ea09ff2ebd       5.55MB
gcr.io/k8s-minikube/storage-provisioner   v1.8.1              4689081edb103       80.8MB
k8s.gcr.io/coredns                        1.6.5               70f311871ae12       41.6MB
k8s.gcr.io/etcd                           3.4.3-0             303ce5db0e90d       288MB
k8s.gcr.io/kube-addon-manager             v9.0.2              bd12a212f9dcb       83.1MB
k8s.gcr.io/kube-apiserver                 v1.17.0             0cae8d5cc64c7       171MB
k8s.gcr.io/kube-controller-manager        v1.17.0             5eb3b74868724       161MB
k8s.gcr.io/kube-proxy                     v1.17.0             7d54289267dc5       116MB
k8s.gcr.io/kube-scheduler                 v1.17.0             78c190f736b11       94.4MB
k8s.gcr.io/pause                          3.1                 da86e6ba6ca19       742kB
kubernetesui/dashboard                    v2.0.0-beta8        eb51a35975256       90.8MB
kubernetesui/metrics-scraper              v1.0.2              3b08661dc379d       40.1MB

```
minikube-bot: All Times minikube: [ 121.107591 126.717778 109.740154]
All Times Minikube (PR 6051): [ 124.533493 123.490417 122.376876]

Average minikube: **119.188508**
Average Minikube (PR 6051): **123.466929**

Averages Time Per Log
```
+----------------------+-----------+--------------------+
|         LOG          | MINIKUBE  | MINIKUBE (PR 6051) |
+----------------------+-----------+--------------------+
| minikube v           |  0.358930 |           0.248033 |
| Creating kvm2        | 47.576860 |          46.622076 |
| Preparing Kubernetes | 46.745881 |          49.990048 |
| Pulling images       |  2.940671 |           3.036284 |
| Launching Kubernetes | 18.869884 |          20.960246 |
| Waiting for cluster  |  2.723588 |           2.564885 |
+----------------------+-----------+--------------------+
```

tstromberg: Resolve the merge conflict and let's get this merged :) 
k8s-ci-robot: @medyagh: PR needs rebase.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
</details>

Patch:
diff --git a/cmd/minikube/cmd/cache.go b/cmd/minikube/cmd/cache.go
index 3952b8185..772e521f1 100644
--- a/cmd/minikube/cmd/cache.go
+++ b/cmd/minikube/cmd/cache.go
@@ -22,6 +22,7 @@ import (
 	"k8s.io/minikube/pkg/minikube/config"
 	"k8s.io/minikube/pkg/minikube/constants"
 	"k8s.io/minikube/pkg/minikube/exit"
+	"k8s.io/minikube/pkg/minikube/image"
 	"k8s.io/minikube/pkg/minikube/localpath"
 	"k8s.io/minikube/pkg/minikube/machine"
 )
@@ -64,12 +65,25 @@ var deleteCacheCmd = &cobra.Command{
 			exit.WithError("Failed to delete images from config", err)
 		}
 		// Delete images from cache/images directory
-		if err := machine.DeleteFromImageCacheDir(args); err != nil {
+		if err := image.DeleteFromCacheDir(args); err != nil {
 			exit.WithError("Failed to delete images", err)
 		}
 	},
 }
 
+// reloadCacheCmd represents the cache reload command
+var reloadCacheCmd = &cobra.Command{
+	Use:   "reload",
+	Short: "reload cached images.",
+	Long:  "reloads images previously added using the 'cache add' subcommand",
+	Run: func(cmd *cobra.Command, args []string) {
+		err := cacheAndLoadImagesInConfig()
+		if err != nil {
+			exit.WithError("Failed to reload cached images", err)
+		}
+	},
+}
+
 func imagesInConfigFile() ([]string, error) {
 	configFile, err := config.ReadConfig(localpath.ConfigFile)
 	if err != nil {
@@ -85,8 +99,9 @@ func imagesInConfigFile() ([]string, error) {
 	return []string{}, nil
 }
 
-// CacheImagesInConfigFile caches the images currently in the config file (minikube start)
-func CacheImagesInConfigFile() error {
+// saveImagesToTarFromConfig saves images to tar in cache which specified in config file.
+// currently only used by download-only option
+func saveImagesToTarFromConfig() error {
 	images, err := imagesInConfigFile()
 	if err != nil {
 		return err
@@ -94,11 +109,12 @@ func CacheImagesInConfigFile() error {
 	if len(images) == 0 {
 		return nil
 	}
-	return machine.CacheImages(images, constants.ImageCacheDir)
+	return image.SaveToDir(images, constants.ImageCacheDir)
 }
 
-// loadCachedImagesInConfigFile loads the images currently in the config file (minikube start)
-func loadCachedImagesInConfigFile() error {
+// cacheAndLoadImagesInConfig loads the images currently in the config file
+// called by 'start' and 'cache reload' commands.
+func cacheAndLoadImagesInConfig() error {
 	images, err := imagesInConfigFile()
 	if err != nil {
 		return err
@@ -112,4 +128,5 @@ func loadCachedImagesInConfigFile() error {
 func init() {
 	cacheCmd.AddCommand(addCacheCmd)
 	cacheCmd.AddCommand(deleteCacheCmd)
+	cacheCmd.AddCommand(reloadCacheCmd)
 }
diff --git a/cmd/minikube/cmd/start.go b/cmd/minikube/cmd/start.go
index 675ec127a..a3ff00398 100644
--- a/cmd/minikube/cmd/start.go
+++ b/cmd/minikube/cmd/start.go
@@ -327,7 +327,7 @@ func runStart(cmd *cobra.Command, args []string) {
 
 	// Now that the ISO is downloaded, pull images in the background while the VM boots.
 	var cacheGroup errgroup.Group
-	beginCacheImages(&cacheGroup, config.KubernetesConfig.ImageRepository, k8sVersion)
+	beginCacheRequiredImages(&cacheGroup, config.KubernetesConfig.ImageRepository, k8sVersion)
 
 	// Abstraction leakage alert: startHost requires the config to be saved, to satistfy pkg/provision/buildroot.
 	// Hence, saveConfig must be called before startHost, and again afterwards when we know the IP.
@@ -342,7 +342,7 @@ func runStart(cmd *cobra.Command, args []string) {
 	// configure the runtime (docker, containerd, crio)
 	cr := configureRuntimes(mRunner, driverName, config.KubernetesConfig)
 	showVersionInfo(k8sVersion, cr)
-	waitCacheImages(&cacheGroup)
+	waitCacheRequiredImages(&cacheGroup)
 
 	// Must be written before bootstrap, otherwise health checks may flake due to stale IP
 	kubeconfig, err := setupKubeconfig(host, &config, config.Name)
@@ -360,7 +360,7 @@ func runStart(cmd *cobra.Command, args []string) {
 	// enable addons with start command
 	enableAddons()
 
-	if err = loadCachedImagesInConfigFile(); err != nil {
+	if err = cacheAndLoadImagesInConfig(); err != nil {
 		out.T(out.FailureType, "Unable to load cached images from config file.")
 	}
 
@@ -460,9 +460,9 @@ func handleDownloadOnly(cacheGroup *errgroup.Group, k8sVersion string) {
 	if err := doCacheBinaries(k8sVersion); err != nil {
 		exit.WithError("Failed to cache binaries", err)
 	}
-	waitCacheImages(cacheGroup)
-	if err := CacheImagesInConfigFile(); err != nil {
-		exit.WithError("Failed to cache images", err)
+	waitCacheRequiredImages(cacheGroup)
+	if err := saveImagesToTarFromConfig(); err != nil {
+		exit.WithError("Failed to cache images to tar", err)
 	}
 	out.T(out.Check, "Download complete!")
 	os.Exit(0)
@@ -820,8 +820,8 @@ func doCacheBinaries(k8sVersion string) error {
 	return machine.CacheBinariesForBootstrapper(k8sVersion, viper.GetString(cmdcfg.Bootstrapper))
 }
 
-// beginCacheImages caches Docker images in the background
-func beginCacheImages(g *errgroup.Group, imageRepository string, k8sVersion string) {
+// beginCacheRequiredImages caches images required for kubernetes version in the background
+func beginCacheRequiredImages(g *errgroup.Group, imageRepository string, k8sVersion string) {
 	if !viper.GetBool(cacheImages) {
 		return
 	}
@@ -831,8 +831,8 @@ func beginCacheImages(g *errgroup.Group, imageRepository string, k8sVersion stri
 	})
 }
 
-// waitCacheImages blocks until the image cache jobs complete
-func waitCacheImages(g *errgroup.Group) {
+// waitCacheRequiredImages blocks until the required images are all cached.
+func waitCacheRequiredImages(g *errgroup.Group) {
 	if !viper.GetBool(cacheImages) {
 		return
 	}
diff --git a/go.mod b/go.mod
index 518eb4df6..af6cb4a10 100644
--- a/go.mod
+++ b/go.mod
@@ -16,7 +16,7 @@ require (
 	github.com/cheggaaa/pb/v3 v3.0.1
 	github.com/cloudfoundry-attic/jibber_jabber v0.0.0-20151120183258-bcc4c8345a21
 	github.com/cloudfoundry/jibber_jabber v0.0.0-20151120183258-bcc4c8345a21 // indirect
-	github.com/docker/docker v1.13.1 // indirect
+	github.com/docker/docker v1.13.1
 	github.com/docker/go-units v0.3.3
 	github.com/docker/machine v0.7.1-0.20190718054102-a555e4f7a8f5 // version is 0.7.1 to pin to a555e4f7a8f5
 	github.com/elazarl/goproxy v0.0.0-20190421051319-9d40249d3c2f
diff --git a/pkg/minikube/image/cache.go b/pkg/minikube/image/cache.go
new file mode 100644
index 000000000..93ac59e59
--- /dev/null
+++ b/pkg/minikube/image/cache.go
@@ -0,0 +1,137 @@
+/*
+Copyright 2016 The Kubernetes Authors All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package image
+
+import (
+	"io/ioutil"
+	"os"
+	"path/filepath"
+	"time"
+
+	"github.com/golang/glog"
+	"github.com/google/go-containerregistry/pkg/name"
+	"github.com/google/go-containerregistry/pkg/v1/tarball"
+	"github.com/pkg/errors"
+	"golang.org/x/sync/errgroup"
+	"k8s.io/minikube/pkg/minikube/constants"
+	"k8s.io/minikube/pkg/minikube/localpath"
+)
+
+// DeleteFromCacheDir deletes tar files stored in cache dir
+func DeleteFromCacheDir(images []string) error {
+	for _, image := range images {
+		path := filepath.Join(constants.ImageCacheDir, image)
+		path = localpath.SanitizeCacheDir(path)
+		glog.Infoln("Deleting image in cache at ", path)
+		if err := os.Remove(path); err != nil {
+			return err
+		}
+	}
+	return cleanImageCacheDir()
+}
+
+// SaveToDir will cache images on the host
+//
+// The cache directory currently caches images using the imagename_tag
+// For example, k8s.gcr.io/kube-addon-manager:v6.5 would be
+// stored at $CACHE_DIR/k8s.gcr.io/kube-addon-manager_v6.5
+func SaveToDir(images []string, cacheDir string) error {
+	var g errgroup.Group
+	for _, image := range images {
+		image := image
+		g.Go(func() error {
+			dst := filepath.Join(cacheDir, image)
+			dst = localpath.SanitizeCacheDir(dst)
+			if err := saveToTarFile(image, dst); err != nil {
+				glog.Errorf("save image to file %q -> %q failed: %v", image, dst, err)
+				return errors.Wrapf(err, "caching image %q", dst)
+			}
+			glog.Infof("save to tar file %s -> %s succeeded", image, dst)
+			return nil
+		})
+	}
+	if err := g.Wait(); err != nil {
+		return errors.Wrap(err, "caching images")
+	}
+	glog.Infoln("Successfully saved all images to host disk.")
+	return nil
+}
+
+// saveToTarFile caches an image
+func saveToTarFile(image, dst string) error {
+	start := time.Now()
+	defer func() {
+		glog.Infof("cache image %q -> %s to local destination -> %q", image, dst, time.Since(start))
+	}()
+
+	if _, err := os.Stat(dst); err == nil {
+		glog.Infof("%s exists", dst)
+		return nil
+	}
+
+	dstPath, err := localpath.DstPath(dst)
+	if err != nil {
+		return errors.Wrap(err, "getting destination path")
+	}
+
+	if err := os.MkdirAll(filepath.Dir(dstPath), 0777); err != nil {
+		return errors.Wrapf(err, "making cache image directory: %s", dst)
+	}
+
+	ref, err := name.ParseReference(image, name.WeakValidation)
+	if err != nil {
+		return errors.Wrapf(err, "parsing image ref name for %s", image)
+	}
+
+	img, err := retrieveImage(ref)
+	if err != nil {
+		glog.Warningf("unable to retrieve image: %v", err)
+	}
+	glog.Infoln("opening: ", dstPath)
+	f, err := ioutil.TempFile(filepath.Dir(dstPath), filepath.Base(dstPath)+".*.tmp")
+	if err != nil {
+		return err
+	}
+	defer func() {
+		// If we left behind a temp file, remove it.
+		_, err := os.Stat(f.Name())
+		if err == nil {
+			err = os.Remove(f.Name())
+			if err != nil {
+				glog.Warningf("failed to clean up the temp file %s: %v", f.Name(), err)
+			}
+		}
+	}()
+	tag, err := name.NewTag(image, name.WeakValidation)
+	if err != nil {
+		return errors.Wrap(err, "newtag")
+	}
+	err = tarball.Write(tag, img, &tarball.WriteOptions{}, f)
+	if err != nil {
+		return errors.Wrap(err, "write")
+	}
+	err = f.Close()
+	if err != nil {
+		return errors.Wrap(err, "close")
+	}
+	err = os.Rename(f.Name(), dstPath)
+	if err != nil {
+		return errors.Wrap(err, "rename")
+	}
+	glog.Infof("%s exists", dst)
+	return nil
+}
diff --git a/pkg/minikube/image/image.go b/pkg/minikube/image/image.go
new file mode 100644
index 000000000..553105e87
--- /dev/null
+++ b/pkg/minikube/image/image.go
@@ -0,0 +1,118 @@
+/*
+Copyright 2016 The Kubernetes Authors All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package image
+
+import (
+	"context"
+	"io/ioutil"
+	"os"
+	"path/filepath"
+	"time"
+
+	"github.com/docker/docker/client"
+	"github.com/golang/glog"
+	"github.com/google/go-containerregistry/pkg/authn"
+	"github.com/google/go-containerregistry/pkg/name"
+	v1 "github.com/google/go-containerregistry/pkg/v1"
+	"github.com/google/go-containerregistry/pkg/v1/daemon"
+	"github.com/google/go-containerregistry/pkg/v1/remote"
+	"k8s.io/minikube/pkg/minikube/constants"
+)
+
+// DigestByDockerLib uses client by docker lib to return image digest
+// img.ID in as same as image digest
+func DigestByDockerLib(imgClient *client.Client, imgName string) string {
+	ctx, cancel := context.WithTimeout(context.Background(), time.Second*10)
+	defer cancel()
+	img, _, err := imgClient.ImageInspectWithRaw(ctx, imgName)
+	if err != nil && !client.IsErrImageNotFound(err) {
+		glog.Infof("couldn't find image digest %s from local daemon: %v ", imgName, err)
+		return ""
+	}
+	return img.ID
+}
+
+// DigestByGoLib gets image digest uses go-containerregistry lib
+// which is 4s slower thabn local daemon per lookup https://github.com/google/go-containerregistry/issues/627
+func DigestByGoLib(imgName string) string {
+	ref, err := name.ParseReference(imgName, name.WeakValidation)
+	if err != nil {
+		glog.Infof("error parsing image name %s ref %v ", imgName, err)
+		return ""
+	}
+
+	img, err := retrieveImage(ref)
+	if err != nil {
+		glog.Infof("error retrieve Image %s ref %v ", imgName, err)
+		return ""
+	}
+
+	cf, err := img.ConfigName()
+	if err != nil {
+		glog.Infof("error getting Image config name %s %v ", imgName, err)
+		return cf.Hex
+	}
+	return cf.Hex
+}
+
+func retrieveImage(ref name.Reference) (v1.Image, error) {
+	glog.Infof("retrieving image: %+v", ref)
+	img, err := daemon.Image(ref)
+	if err == nil {
+		glog.Infof("found %s locally: %+v", ref.Name(), img)
+		return img, nil
+	}
+	// reference does not exist in the local daemon
+	if err != nil {
+		glog.Infof("daemon lookup for %+v: %v", ref, err)
+	}
+
+	img, err = remote.Image(ref, remote.WithAuthFromKeychain(authn.DefaultKeychain))
+	if err == nil {
+		return img, nil
+	}
+
+	glog.Warningf("authn lookup for %+v (trying anon): %+v", ref, err)
+	img, err = remote.Image(ref)
+	return img, err
+}
+
+func cleanImageCacheDir() error {
+	err := filepath.Walk(constants.ImageCacheDir, func(path string, info os.FileInfo, err error) error {
+		// If error is not nil, it's because the path was already deleted and doesn't exist
+		// Move on to next path
+		if err != nil {
+			return nil
+		}
+		// Check if path is directory
+		if !info.IsDir() {
+			return nil
+		}
+		// If directory is empty, delete it
+		entries, err := ioutil.ReadDir(path)
+		if err != nil {
+			return err
+		}
+		if len(entries) == 0 {
+			if err = os.Remove(path); err != nil {
+				return err
+			}
+		}
+		return nil
+	})
+	return err
+}
diff --git a/pkg/minikube/localpath/localpath.go b/pkg/minikube/localpath/localpath.go
index 820d21f1d..b5a00092c 100644
--- a/pkg/minikube/localpath/localpath.go
+++ b/pkg/minikube/localpath/localpath.go
@@ -18,8 +18,13 @@ package localpath
 
 import (
 	"os"
+	"os/exec"
 	"path/filepath"
+	"runtime"
+	"strings"
 
+	"github.com/golang/glog"
+	"github.com/pkg/errors"
 	"k8s.io/client-go/util/homedir"
 )
 
@@ -55,3 +60,82 @@ func MachinePath(machine string, miniHome ...string) string {
 	}
 	return filepath.Join(miniPath, "machines", machine)
 }
+
+// SanitizeCacheDir
+// # ParseReference cannot have a : in the directory path
+func SanitizeCacheDir(image string) string {
+	if runtime.GOOS == "windows" && hasWindowsDriveLetter(image) {
+		// not sanitize Windows drive letter.
+		s := image[:2] + strings.Replace(image[2:], ":", "_", -1)
+		glog.Infof("windows sanitize: %s -> %s", image, s)
+		return s
+	}
+	return strings.Replace(image, ":", "_", -1)
+}
+
+func hasWindowsDriveLetter(s string) bool {
+	if len(s) < 3 {
+		return false
+	}
+
+	drive := s[:3]
+	for _, b := range "CDEFGHIJKLMNOPQRSTUVWXYZABcdefghijklmnopqrstuvwxyzab" {
+		if d := string(b) + ":"; drive == d+`\` || drive == d+`/` {
+			return true
+		}
+	}
+
+	return false
+}
+
+// DstPath returns an os specific
+func DstPath(dst string) (string, error) {
+	if runtime.GOOS == "windows" && hasWindowsDriveLetter(dst) {
+		// ParseReference does not support a Windows drive letter.
+		// Therefore, will replace the drive letter to a volume name.
+		var err error
+		if dst, err = replaceWinDriveLetterToVolumeName(dst); err != nil {
+			return "", errors.Wrap(err, "parsing docker archive dst ref: replace a Win drive letter to a volume name")
+		}
+	}
+	return dst, nil
+}
+
+// Replace a drive letter to a volume name.
+func replaceWinDriveLetterToVolumeName(s string) (string, error) {
+	vname, err := getWindowsVolumeName(s[:1])
+	if err != nil {
+		return "", err
+	}
+	path := vname + s[3:]
+
+	return path, nil
+}
+
+func getWindowsVolumeNameCmd(d string) (string, error) {
+	cmd := exec.Command("wmic", "volume", "where", "DriveLetter = '"+d+":'", "get", "DeviceID")
+
+	stdout, err := cmd.Output()
+	if err != nil {
+		return "", err
+	}
+
+	outs := strings.Split(strings.Replace(string(stdout), "\r", "", -1), "\n")
+
+	var vname string
+	for _, l := range outs {
+		s := strings.TrimSpace(l)
+		if strings.HasPrefix(s, `\\?\Volume{`) && strings.HasSuffix(s, `}\`) {
+			vname = s
+			break
+		}
+	}
+
+	if vname == "" {
+		return "", errors.New("failed to get a volume GUID")
+	}
+
+	return vname, nil
+}
+
+var getWindowsVolumeName = getWindowsVolumeNameCmd
diff --git a/pkg/minikube/machine/cache_binaries.go b/pkg/minikube/machine/cache_binaries.go
index 98b7824d8..86102647f 100644
--- a/pkg/minikube/machine/cache_binaries.go
+++ b/pkg/minikube/machine/cache_binaries.go
@@ -43,7 +43,7 @@ func CacheBinariesForBootstrapper(version string, clusterBootstrapper string) er
 		bin := bin // https://golang.org/doc/faq#closures_and_goroutines
 		g.Go(func() error {
 			if _, err := CacheBinary(bin, version, "linux", runtime.GOARCH); err != nil {
-				return errors.Wrapf(err, "caching image %s", bin)
+				return errors.Wrapf(err, "caching binary %s", bin)
 			}
 			return nil
 		})
diff --git a/pkg/minikube/machine/cache_images.go b/pkg/minikube/machine/cache_images.go
index 1ac409b45..599270c90 100644
--- a/pkg/minikube/machine/cache_images.go
+++ b/pkg/minikube/machine/cache_images.go
@@ -18,24 +18,14 @@ package machine
 
 import (
 	"fmt"
-	"io/ioutil"
 	"os"
-	"os/exec"
 	"path"
 	"path/filepath"
-	"runtime"
-	"strings"
 	"sync"
-	"time"
 
+	"github.com/docker/docker/client"
 	"github.com/docker/machine/libmachine/state"
 	"github.com/golang/glog"
-	"github.com/google/go-containerregistry/pkg/authn"
-	"github.com/google/go-containerregistry/pkg/name"
-	v1 "github.com/google/go-containerregistry/pkg/v1"
-	"github.com/google/go-containerregistry/pkg/v1/daemon"
-	"github.com/google/go-containerregistry/pkg/v1/remote"
-	"github.com/google/go-containerregistry/pkg/v1/tarball"
 	"github.com/pkg/errors"
 	"golang.org/x/sync/errgroup"
 	"k8s.io/minikube/pkg/minikube/assets"
@@ -45,14 +35,14 @@ import (
 	"k8s.io/minikube/pkg/minikube/config"
 	"k8s.io/minikube/pkg/minikube/constants"
 	"k8s.io/minikube/pkg/minikube/cruntime"
+	"k8s.io/minikube/pkg/minikube/image"
+	"k8s.io/minikube/pkg/minikube/localpath"
 	"k8s.io/minikube/pkg/minikube/vmpath"
 )
 
 // loadRoot is where images should be loaded from within the guest VM
 var loadRoot = path.Join(vmpath.GuestPersistentDir, "images")
 
-var getWindowsVolumeName = getWindowsVolumeNameCmd
-
 // loadImageLock is used to serialize image loads to avoid overloading the guest VM
 var loadImageLock sync.Mutex
 
@@ -63,40 +53,13 @@ func CacheImagesForBootstrapper(imageRepository string, version string, clusterB
 		return errors.Wrap(err, "cached images list")
 	}
 
-	if err := CacheImages(images, constants.ImageCacheDir); err != nil {
+	if err := image.SaveToDir(images, constants.ImageCacheDir); err != nil {
 		return errors.Wrapf(err, "Caching images for %s", clusterBootstrapper)
 	}
 
 	return nil
 }
 
-// CacheImages will cache images on the host
-//
-// The cache directory currently caches images using the imagename_tag
-// For example, k8s.gcr.io/kube-addon-manager:v6.5 would be
-// stored at $CACHE_DIR/k8s.gcr.io/kube-addon-manager_v6.5
-func CacheImages(images []string, cacheDir string) error {
-	var g errgroup.Group
-	for _, image := range images {
-		image := image
-		g.Go(func() error {
-			dst := filepath.Join(cacheDir, image)
-			dst = sanitizeCacheDir(dst)
-			if err := CacheImage(image, dst); err != nil {
-				glog.Errorf("CacheImage %s -> %s failed: %v", image, dst, err)
-				return errors.Wrapf(err, "caching image %s", dst)
-			}
-			glog.Infof("CacheImage %s -> %s succeeded", image, dst)
-			return nil
-		})
-	}
-	if err := g.Wait(); err != nil {
-		return errors.Wrap(err, "caching images")
-	}
-	glog.Infoln("Successfully cached all images.")
-	return nil
-}
-
 // LoadImages loads previously cached images into the container runtime
 func LoadImages(cc *config.MachineConfig, runner command.Runner, images []string, cacheDir string) error {
 	glog.Infof("LoadImages start: %s", images)
@@ -107,10 +70,16 @@ func LoadImages(cc *config.MachineConfig, runner command.Runner, images []string
 		return errors.Wrap(err, "runtime")
 	}
 
+	imgClient, err := client.NewEnvClient() // image client
+	if err != nil {
+		glog.Infof("couldn't get a local image daemon which might be ok: %v", err)
+		imgClient = nil
+	}
+
 	for _, image := range images {
 		image := image
 		g.Go(func() error {
-			err := needsTransfer(image, cr)
+			err := needsTransfer(imgClient, image, cr)
 			if err == nil {
 				return nil
 			}
@@ -126,31 +95,31 @@ func LoadImages(cc *config.MachineConfig, runner command.Runner, images []string
 }
 
 // needsTransfer returns an error if an image needs to be retransfered
-func needsTransfer(image string, cr cruntime.Manager) error {
-	ref, err := name.ParseReference(image, name.WeakValidation)
-	if err != nil {
-		return errors.Wrap(err, "parse ref")
-	}
-
-	img, err := retrieveImage(ref)
-	if err != nil {
-		return errors.Wrap(err, "retrieve")
+func needsTransfer(imgClient *client.Client, imgName string, cr cruntime.Manager) error {
+	imgDgst := ""         // for instance sha256:7c92a2c6bbcb6b6beff92d0a940779769c2477b807c202954c537e2e0deb9bed
+	if imgClient != nil { // if possible try to get img digest from Client lib which is 4s faster.
+		imgDgst = image.DigestByDockerLib(imgClient, imgName)
+		if imgDgst != "" {
+			if !cr.ImageExists(imgName, imgDgst) {
+				return fmt.Errorf("%q does not exist at hash %q in container runtime", imgName, imgDgst)
+			}
+			return nil
+		}
 	}
-
-	cf, err := img.ConfigName()
-	if err != nil {
-		return errors.Wrap(err, "image hash")
+	// if not found with method above try go-container lib (which is 4s slower)
+	imgDgst = image.DigestByGoLib(imgName)
+	if imgDgst == "" {
+		return fmt.Errorf("got empty img digest %q for %s", imgDgst, imgName)
 	}
-
-	if !cr.ImageExists(image, cf.Hex) {
-		return fmt.Errorf("%q does not exist at hash %q in container runtime", image, cf.Hex)
+	if !cr.ImageExists(imgName, imgDgst) {
+		return fmt.Errorf("%q does not exist at hash %q in container runtime", imgName, imgDgst)
 	}
 	return nil
 }
 
 // CacheAndLoadImages caches and loads images to all profiles
 func CacheAndLoadImages(images []string) error {
-	if err := CacheImages(images, constants.ImageCacheDir); err != nil {
+	if err := image.SaveToDir(images, constants.ImageCacheDir); err != nil {
 		return err
 	}
 	api, err := NewAPIClient()
@@ -162,7 +131,7 @@ func CacheAndLoadImages(images []string) error {
 	if err != nil {
 		return errors.Wrap(err, "list profiles")
 	}
-	for _, p := range profiles { // adding images to all the profiles
+	for _, p := range profiles { // loading images to all running profiles
 		pName := p.Name // capture the loop variable
 		status, err := cluster.GetHostStatus(api, pName)
 		if err != nil {
@@ -192,69 +161,6 @@ func CacheAndLoadImages(images []string) error {
 	return err
 }
 
-// # ParseReference cannot have a : in the directory path
-func sanitizeCacheDir(image string) string {
-	if runtime.GOOS == "windows" && hasWindowsDriveLetter(image) {
-		// not sanitize Windows drive letter.
-		s := image[:2] + strings.Replace(image[2:], ":", "_", -1)
-		glog.Infof("windows sanitize: %s -> %s", image, s)
-		return s
-	}
-	return strings.Replace(image, ":", "_", -1)
-}
-
-func hasWindowsDriveLetter(s string) bool {
-	if len(s) < 3 {
-		return false
-	}
-
-	drive := s[:3]
-	for _, b := range "CDEFGHIJKLMNOPQRSTUVWXYZABcdefghijklmnopqrstuvwxyzab" {
-		if d := string(b) + ":"; drive == d+`\` || drive == d+`/` {
-			return true
-		}
-	}
-
-	return false
-}
-
-// Replace a drive letter to a volume name.
-func replaceWinDriveLetterToVolumeName(s string) (string, error) {
-	vname, err := getWindowsVolumeName(s[:1])
-	if err != nil {
-		return "", err
-	}
-	path := vname + s[3:]
-
-	return path, nil
-}
-
-func getWindowsVolumeNameCmd(d string) (string, error) {
-	cmd := exec.Command("wmic", "volume", "where", "DriveLetter = '"+d+":'", "get", "DeviceID")
-
-	stdout, err := cmd.Output()
-	if err != nil {
-		return "", err
-	}
-
-	outs := strings.Split(strings.Replace(string(stdout), "\r", "", -1), "\n")
-
-	var vname string
-	for _, l := range outs {
-		s := strings.TrimSpace(l)
-		if strings.HasPrefix(s, `\\?\Volume{`) && strings.HasSuffix(s, `}\`) {
-			vname = s
-			break
-		}
-	}
-
-	if vname == "" {
-		return "", errors.New("failed to get a volume GUID")
-	}
-
-	return vname, nil
-}
-
 // transferAndLoadImage transfers and loads a single image from the cache
 func transferAndLoadImage(cr command.Runner, k8s config.KubernetesConfig, imgName string, cacheDir string) error {
 	r, err := cruntime.New(cruntime.Config{Type: k8s.ContainerRuntime, Runner: cr})
@@ -262,7 +168,7 @@ func transferAndLoadImage(cr command.Runner, k8s config.KubernetesConfig, imgNam
 		return errors.Wrap(err, "runtime")
 	}
 	src := filepath.Join(cacheDir, imgName)
-	src = sanitizeCacheDir(src)
+	src = localpath.SanitizeCacheDir(src)
 	glog.Infof("Loading image from cache: %s", src)
 	filename := filepath.Base(src)
 	if _, err := os.Stat(src); err != nil {
@@ -288,144 +194,3 @@ func transferAndLoadImage(cr command.Runner, k8s config.KubernetesConfig, imgNam
 	glog.Infof("Transferred and loaded %s from cache", src)
 	return nil
 }
-
-// DeleteFromImageCacheDir deletes images from the cache
-func DeleteFromImageCacheDir(images []string) error {
-	for _, image := range images {
-		path := filepath.Join(constants.ImageCacheDir, image)
-		path = sanitizeCacheDir(path)
-		glog.Infoln("Deleting image in cache at ", path)
-		if err := os.Remove(path); err != nil {
-			return err
-		}
-	}
-	return cleanImageCacheDir()
-}
-
-func cleanImageCacheDir() error {
-	err := filepath.Walk(constants.ImageCacheDir, func(path string, info os.FileInfo, err error) error {
-		// If error is not nil, it's because the path was already deleted and doesn't exist
-		// Move on to next path
-		if err != nil {
-			return nil
-		}
-		// Check if path is directory
-		if !info.IsDir() {
-			return nil
-		}
-		// If directory is empty, delete it
-		entries, err := ioutil.ReadDir(path)
-		if err != nil {
-			return err
-		}
-		if len(entries) == 0 {
-			if err = os.Remove(path); err != nil {
-				return err
-			}
-		}
-		return nil
-	})
-	return err
-}
-
-func getDstPath(dst string) (string, error) {
-	if runtime.GOOS == "windows" && hasWindowsDriveLetter(dst) {
-		// ParseReference does not support a Windows drive letter.
-		// Therefore, will replace the drive letter to a volume name.
-		var err error
-		if dst, err = replaceWinDriveLetterToVolumeName(dst); err != nil {
-			return "", errors.Wrap(err, "parsing docker archive dst ref: replace a Win drive letter to a volume name")
-		}
-	}
-
-	return dst, nil
-}
-
-// CacheImage caches an image
-func CacheImage(image, dst string) error {
-	start := time.Now()
-	glog.Infof("CacheImage: %s -> %s", image, dst)
-	defer func() {
-		glog.Infof("CacheImage: %s -> %s completed in %s", image, dst, time.Since(start))
-	}()
-
-	if _, err := os.Stat(dst); err == nil {
-		glog.Infof("%s exists", dst)
-		return nil
-	}
-
-	dstPath, err := getDstPath(dst)
-	if err != nil {
-		return errors.Wrap(err, "getting destination path")
-	}
-
-	if err := os.MkdirAll(filepath.Dir(dstPath), 0777); err != nil {
-		return errors.Wrapf(err, "making cache image directory: %s", dst)
-	}
-
-	ref, err := name.ParseReference(image, name.WeakValidation)
-	if err != nil {
-		return errors.Wrap(err, "creating docker image name")
-	}
-
-	img, err := retrieveImage(ref)
-	if err != nil {
-		glog.Warningf("unable to retrieve image: %v", err)
-	}
-
-	glog.Infoln("OPENING: ", dstPath)
-	f, err := ioutil.TempFile(filepath.Dir(dstPath), filepath.Base(dstPath)+".*.tmp")
-	if err != nil {
-		return err
-	}
-	defer func() {
-		// If we left behind a temp file, remove it.
-		_, err := os.Stat(f.Name())
-		if err == nil {
-			os.Remove(f.Name())
-			if err != nil {
-				glog.Warningf("Failed to clean up the temp file %s: %v", f.Name(), err)
-			}
-		}
-	}()
-	tag, err := name.NewTag(image, name.WeakValidation)
-	if err != nil {
-		return errors.Wrap(err, "newtag")
-	}
-	err = tarball.Write(tag, img, &tarball.WriteOptions{}, f)
-	if err != nil {
-		return errors.Wrap(err, "write")
-	}
-	err = f.Close()
-	if err != nil {
-		return errors.Wrap(err, "close")
-	}
-	err = os.Rename(f.Name(), dstPath)
-	if err != nil {
-		return errors.Wrap(err, "rename")
-	}
-	glog.Infof("%s exists", dst)
-	return nil
-}
-
-func retrieveImage(ref name.Reference) (v1.Image, error) {
-	glog.Infof("retrieving image: %+v", ref)
-	img, err := daemon.Image(ref)
-	if err == nil {
-		glog.Infof("found %s locally: %+v", ref.Name(), img)
-		return img, nil
-	}
-	// reference does not exist in the local daemon
-	if err != nil {
-		glog.Infof("daemon lookup for %+v: %v", ref, err)
-	}
-
-	img, err = remote.Image(ref, remote.WithAuthFromKeychain(authn.DefaultKeychain))
-	if err == nil {
-		return img, nil
-	}
-
-	glog.Warningf("authn lookup for %+v (trying anon): %+v", ref, err)
-	img, err = remote.Image(ref)
-	return img, err
-}


Test patch:
diff --git a/pkg/minikube/machine/cache_images_test.go b/pkg/minikube/localpath/localpath_test.go
similarity index 98%
rename from pkg/minikube/machine/cache_images_test.go
rename to pkg/minikube/localpath/localpath_test.go
index 6cf9ead4a..173ca5df8 100644
--- a/pkg/minikube/machine/cache_images_test.go
+++ b/pkg/minikube/localpath/localpath_test.go
@@ -14,7 +14,7 @@ See the License for the specific language governing permissions and
 limitations under the License.
 */
 
-package machine
+package localpath
 
 import (
 	"io/ioutil"
diff --git a/test/integration/functional_test.go b/test/integration/functional_test.go
index a21ec2a56..1480538ca 100644
--- a/test/integration/functional_test.go
+++ b/test/integration/functional_test.go
@@ -323,7 +323,7 @@ func validateCacheCmd(ctx context.Context, t *testing.T, profile string) {
 				}
 			}
 		})
-		t.Run("delete", func(t *testing.T) {
+		t.Run("delete busybox:1.28.4-glibc", func(t *testing.T) {
 			_, err := Run(t, exec.CommandContext(ctx, Target(), "cache", "delete", "busybox:1.28.4-glibc"))
 			if err != nil {
 				t.Errorf("failed to delete image busybox:1.28.4-glibc from cache: %v", err)
@@ -346,13 +346,38 @@ func validateCacheCmd(ctx context.Context, t *testing.T, profile string) {
 		t.Run("verify cache inside node", func(t *testing.T) {
 			rr, err := Run(t, exec.CommandContext(ctx, Target(), "-p", profile, "ssh", "sudo", "crictl", "images"))
 			if err != nil {
-				t.Errorf("failed to get docker images through ssh %v", err)
+				t.Errorf("failed to get images by %q ssh %v", rr.Command(), err)
 			}
 			if !strings.Contains(rr.Output(), "1.28.4-glibc") {
 				t.Errorf("expected '1.28.4-glibc' to be in the output: %s", rr.Output())
 			}
 
 		})
+
+		t.Run("cache reload", func(t *testing.T) { // deleting image inside minikube node manually and expecting reload to bring it back
+			img := "busybox:latest"
+			// deleting image inside minikube node manually
+			rr, err := Run(t, exec.CommandContext(ctx, Target(), "-p", profile, "ssh", "sudo", "docker", "rmi", img)) // for some reason crictl rmi doesn't work
+			if err != nil {
+				t.Errorf("failed to delete inside the node %q : %v", rr.Command(), err)
+			}
+			// make sure the image is deleted.
+			rr, err = Run(t, exec.CommandContext(ctx, Target(), "-p", profile, "ssh", "sudo", "crictl", "inspecti", img))
+			if err == nil {
+				t.Errorf("expected the image be deleted and get  error but got nil error ! cmd: %q", rr.Command())
+			}
+			// minikube cache reload.
+			rr, err = Run(t, exec.CommandContext(ctx, Target(), "-p", profile, "cache", "reload"))
+			if err != nil {
+				t.Errorf("expected %q to run successfully but got error %v", rr.Command(), err)
+			}
+			// make sure 'cache reload' brought back the manually deleted image.
+			rr, err = Run(t, exec.CommandContext(ctx, Target(), "-p", profile, "ssh", "sudo", "crictl", "inspecti", img))
+			if err != nil {
+				t.Errorf("expected to get no error for %q but got %v", rr.Command(), err)
+			}
+		})
+
 	})
 }