Repository: kubernetes/minikube
Pull request #17314 (2023-09-27 20:39:55)

new flag -"-gpu" to enable Nvidia container runtime
Rework of https://github.com/kubernetes/minikube/pull/17287 that removes the `nvidia-docker` container-runtime and uses the `--gpus` flag instead.

```
$ minikube start --gpus all
😄  minikube v1.31.2 on Debian rodete
✨  Using the docker driver based on user configuration
📌  Using Docker driver with root privileges
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
🔥  Creating docker container (CPUs=2, Memory=32100MB) ...
❗  Using GPUs with the Docker driver is experimental, if you experience any issues please report them at: https://github.com/kubernetes/minikube/issues/new/choose
🛠️   Installing the NVIDIA Container Toolkit...
🐳  Preparing Kubernetes v1.28.2 on Docker 24.0.6 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔗  Configuring bridge CNI (Container Networking Interface) ...
🔎  Verifying Kubernetes components...
    ▪ Using image nvcr.io/nvidia/k8s-device-plugin:v0.14.1
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, nvidia-device-plugin, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

$ cat << EOF | kubectl create -f -
apiVersion: v1
kind: Pod
metadata:
  name: nvidia-version-check
spec:
  restartPolicy: OnFailure
  containers:
  - name: nvidia-version-check
    image: "nvidia/cuda:11.0.3-base-ubuntu20.04"
    command: ["nvidia-smi"]
    resources:
      limits:
         nvidia.com/gpu: "1"
EOF
pod/nvidia-version-check created

$ kubectl logs nvidia-version-check
Fri Sep 22 18:45:31 2023       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.125.06   Driver Version: 525.125.06   CUDA Version: 12.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Quadro P1000        On   | 00000000:65:00.0 Off |                  N/A |
| 34%   26C    P8    N/A /  47W |     15MiB /  4096MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
+-----------------------------------------------------------------------------+

$ minikube start --driver kvm --gpus all
😄  minikube v1.31.2 on Debian rodete (kvm/amd64)
✨  Using the kvm2 driver based on user configuration

❌  Exiting due to MK_USAGE: The gpus flag can only be used with the docker driver and docker container-runtime

$ minikube start --gpus cat
😄  minikube v1.31.2 on Debian rodete
✨  Automatically selected the docker driver

❌  Exiting due to MK_USAGE: The gpus flag must be passed a value of "nvidia" or "all"
```

<img width="816" alt="Screenshot 2023-09-27 at 1 40 51 PM" src="https://github.com/kubernetes/minikube/assets/44844360/794009ea-ec79-41fc-9a37-9245adc60763">


k8s-ci-robot: [APPROVALNOTIFIER] This PR is **APPROVED**

This pull-request has been approved by: *<a href="https://github.com/kubernetes/minikube/pull/17314#" title="Author self-approved">spowelljr</a>*

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=kubernetes%2Fminikube).

The pull request process is described [here](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process)

<details >
Needs approval from an approver in each of these files:

- ~~[OWNERS](https://github.com/kubernetes/minikube/blob/master/OWNERS)~~ [spowelljr]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->
medyagh: /ok-to-test
minikube-pr-bot: **kvm2 driver with docker runtime**
```
+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 17314) |
+----------------+----------+---------------------+
| minikube start | 49.7s    | 49.9s               |
| enable ingress | 27.2s    | 28.4s               |
+----------------+----------+---------------------+
```

<details>

Times for minikube start: 48.3s 50.4s 50.0s 48.6s 51.1s 
Times for minikube (PR 17314) start: 51.0s 47.7s 51.0s 51.5s 48.4s 

Times for minikube ingress: 27.7s 25.2s 28.2s 28.1s 27.1s 
Times for minikube (PR 17314) ingress: 28.6s 28.1s 28.7s 28.7s 28.1s 

</details>

**docker driver with docker runtime**
```
+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 17314) |
+----------------+----------+---------------------+
| minikube start | 24.3s    | 22.8s               |
| enable ingress | 21.3s    | 21.7s               |
+----------------+----------+---------------------+
```

<details>

Times for minikube start: 22.3s 26.0s 25.7s 22.2s 25.1s 
Times for minikube (PR 17314) start: 25.6s 22.5s 22.2s 22.1s 21.8s 

Times for minikube ingress: 20.9s 21.9s 20.9s 21.9s 20.9s 
Times for minikube (PR 17314) ingress: 20.9s 21.8s 22.8s 21.8s 20.9s 

</details>

**docker driver with containerd runtime**
```
+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 17314) |
+----------------+----------+---------------------+
| minikube start | 23.5s    | 23.2s               |
| enable ingress | 31.8s    | 29.3s               |
+----------------+----------+---------------------+
```

<details>

Times for minikube start: 23.2s 24.0s 23.9s 23.6s 23.1s 
Times for minikube (PR 17314) start: 24.9s 24.0s 23.3s 23.0s 20.7s 

Times for minikube ingress: 31.3s 31.4s 31.3s 31.4s 33.4s 
Times for minikube (PR 17314) ingress: 31.4s 31.4s 31.3s 32.3s 20.3s 

</details>
minikube-pr-bot: These are the flake rates of all failed tests.
|Environment|Failed Tests|Flake Rate (%)|
|---|---|---|
|[QEMU_macOS](https://gopogh-server-tts3vkcpgq-uc.a.run.app/?env=QEMU_macOS)|TestAddons/parallel/NvidiaDevicePlugin ([gopogh](https://storage.googleapis.com/minikube-builds/logs/17314/31280/QEMU_macOS.html#fail_TestAddons/parallel/NvidiaDevicePlugin))|n/a|
|[KVM_Linux](https://gopogh-server-tts3vkcpgq-uc.a.run.app/?env=KVM_Linux)|TestPreload ([gopogh](https://storage.googleapis.com/minikube-builds/logs/17314/31280/KVM_Linux.html#fail_TestPreload))|0.73 ([chart](https://gopogh-server-tts3vkcpgq-uc.a.run.app/?env=KVM_Linux&test=TestPreload))|
|[Docker_Linux_crio](https://gopogh-server-tts3vkcpgq-uc.a.run.app/?env=Docker_Linux_crio)|TestFunctional/parallel/ImageCommands/ImageTagAndLoadDaemon ([gopogh](https://storage.googleapis.com/minikube-builds/logs/17314/31280/Docker_Linux_crio.html#fail_TestFunctional/parallel/ImageCommands/ImageTagAndLoadDaemon))|0.74 ([chart](https://gopogh-server-tts3vkcpgq-uc.a.run.app/?env=Docker_Linux_crio&test=TestFunctional/parallel/ImageCommands/ImageTagAndLoadDaemon))|
|[Docker_macOS](https://gopogh-server-tts3vkcpgq-uc.a.run.app/?env=Docker_macOS)|TestMountStart/serial/VerifyMountSecond ([gopogh](https://storage.googleapis.com/minikube-builds/logs/17314/31280/Docker_macOS.html#fail_TestMountStart/serial/VerifyMountSecond))|0.86 ([chart](https://gopogh-server-tts3vkcpgq-uc.a.run.app/?env=Docker_macOS&test=TestMountStart/serial/VerifyMountSecond))|
|[KVM_Linux](https://gopogh-server-tts3vkcpgq-uc.a.run.app/?env=KVM_Linux)|TestNoKubernetes/serial/ProfileList ([gopogh](https://storage.googleapis.com/minikube-builds/logs/17314/31280/KVM_Linux.html#fail_TestNoKubernetes/serial/ProfileList))|1.46 ([chart](https://gopogh-server-tts3vkcpgq-uc.a.run.app/?env=KVM_Linux&test=TestNoKubernetes/serial/ProfileList))|
|[Docker_Linux_docker_arm64](https://gopogh-server-tts3vkcpgq-uc.a.run.app/?env=Docker_Linux_docker_arm64)|TestScheduledStopUnix ([gopogh](https://storage.googleapis.com/minikube-builds/logs/17314/31280/Docker_Linux_docker_arm64.html#fail_TestScheduledStopUnix))|1.48 ([chart](https://gopogh-server-tts3vkcpgq-uc.a.run.app/?env=Docker_Linux_docker_arm64&test=TestScheduledStopUnix))|
|[QEMU_macOS](https://gopogh-server-tts3vkcpgq-uc.a.run.app/?env=QEMU_macOS)|TestAddons/serial/GCPAuth/Namespaces ([gopogh](https://storage.googleapis.com/minikube-builds/logs/17314/31280/QEMU_macOS.html#fail_TestAddons/serial/GCPAuth/Namespaces))|3.57 ([chart](https://gopogh-server-tts3vkcpgq-uc.a.run.app/?env=QEMU_macOS&test=TestAddons/serial/GCPAuth/Namespaces))|
|[Docker_macOS](https://gopogh-server-tts3vkcpgq-uc.a.run.app/?env=Docker_macOS)|TestCertOptions ([gopogh](https://storage.googleapis.com/minikube-builds/logs/17314/31280/Docker_macOS.html#fail_TestCertOptions))|5.08 ([chart](https://gopogh-server-tts3vkcpgq-uc.a.run.app/?env=Docker_macOS&test=TestCertOptions))|
|[Docker_macOS](https://gopogh-server-tts3vkcpgq-uc.a.run.app/?env=Docker_macOS)|TestDockerFlags ([gopogh](https://storage.googleapis.com/minikube-builds/logs/17314/31280/Docker_macOS.html#fail_TestDockerFlags))|5.08 ([chart](https://gopogh-server-tts3vkcpgq-uc.a.run.app/?env=Docker_macOS&test=TestDockerFlags))|
|[Docker_macOS](https://gopogh-server-tts3vkcpgq-uc.a.run.app/?env=Docker_macOS)|TestForceSystemdEnv ([gopogh](https://storage.googleapis.com/minikube-builds/logs/17314/31280/Docker_macOS.html#fail_TestForceSystemdEnv))|5.08 ([chart](https://gopogh-server-tts3vkcpgq-uc.a.run.app/?env=Docker_macOS&test=TestForceSystemdEnv))|
|[Docker_macOS](https://gopogh-server-tts3vkcpgq-uc.a.run.app/?env=Docker_macOS)|TestForceSystemdFlag ([gopogh](https://storage.googleapis.com/minikube-builds/logs/17314/31280/Docker_macOS.html#fail_TestForceSystemdFlag))|5.08 ([chart](https://gopogh-server-tts3vkcpgq-uc.a.run.app/?env=Docker_macOS&test=TestForceSystemdFlag))|
|[Docker_macOS](https://gopogh-server-tts3vkcpgq-uc.a.run.app/?env=Docker_macOS)|TestInsufficientStorage ([gopogh](https://storage.googleapis.com/minikube-builds/logs/17314/31280/Docker_macOS.html#fail_TestInsufficientStorage))|5.08 ([chart](https://gopogh-server-tts3vkcpgq-uc.a.run.app/?env=Docker_macOS&test=TestInsufficientStorage))|
|[Docker_macOS](https://gopogh-server-tts3vkcpgq-uc.a.run.app/?env=Docker_macOS)|TestMultiNode/serial/AddNode ([gopogh](https://storage.googleapis.com/minikube-builds/logs/17314/31280/Docker_macOS.html#fail_TestMultiNode/serial/AddNode))|5.08 ([chart](https://gopogh-server-tts3vkcpgq-uc.a.run.app/?env=Docker_macOS&test=TestMultiNode/serial/AddNode))|
|[Docker_macOS](https://gopogh-server-tts3vkcpgq-uc.a.run.app/?env=Docker_macOS)|TestMultiNode/serial/CopyFile ([gopogh](https://storage.googleapis.com/minikube-builds/logs/17314/31280/Docker_macOS.html#fail_TestMultiNode/serial/CopyFile))|5.08 ([chart](https://gopogh-server-tts3vkcpgq-uc.a.run.app/?env=Docker_macOS&test=TestMultiNode/serial/CopyFile))|
|[Docker_macOS](https://gopogh-server-tts3vkcpgq-uc.a.run.app/?env=Docker_macOS)|TestMultiNode/serial/DeleteNode ([gopogh](https://storage.googleapis.com/minikube-builds/logs/17314/31280/Docker_macOS.html#fail_TestMultiNode/serial/DeleteNode))|5.08 ([chart](https://gopogh-server-tts3vkcpgq-uc.a.run.app/?env=Docker_macOS&test=TestMultiNode/serial/DeleteNode))|
|[Docker_macOS](https://gopogh-server-tts3vkcpgq-uc.a.run.app/?env=Docker_macOS)|TestMultiNode/serial/DeployApp2Nodes ([gopogh](https://storage.googleapis.com/minikube-builds/logs/17314/31280/Docker_macOS.html#fail_TestMultiNode/serial/DeployApp2Nodes))|5.08 ([chart](https://gopogh-server-tts3vkcpgq-uc.a.run.app/?env=Docker_macOS&test=TestMultiNode/serial/DeployApp2Nodes))|
|[Docker_macOS](https://gopogh-server-tts3vkcpgq-uc.a.run.app/?env=Docker_macOS)|TestMultiNode/serial/FreshStart2Nodes ([gopogh](https://storage.googleapis.com/minikube-builds/logs/17314/31280/Docker_macOS.html#fail_TestMultiNode/serial/FreshStart2Nodes))|5.08 ([chart](https://gopogh-server-tts3vkcpgq-uc.a.run.app/?env=Docker_macOS&test=TestMultiNode/serial/FreshStart2Nodes))|
|[Docker_macOS](https://gopogh-server-tts3vkcpgq-uc.a.run.app/?env=Docker_macOS)|TestMultiNode/serial/PingHostFrom2Pods ([gopogh](https://storage.googleapis.com/minikube-builds/logs/17314/31280/Docker_macOS.html#fail_TestMultiNode/serial/PingHostFrom2Pods))|5.08 ([chart](https://gopogh-server-tts3vkcpgq-uc.a.run.app/?env=Docker_macOS&test=TestMultiNode/serial/PingHostFrom2Pods))|
|[Docker_macOS](https://gopogh-server-tts3vkcpgq-uc.a.run.app/?env=Docker_macOS)|TestMultiNode/serial/ProfileList ([gopogh](https://storage.googleapis.com/minikube-builds/logs/17314/31280/Docker_macOS.html#fail_TestMultiNode/serial/ProfileList))|5.08 ([chart](https://gopogh-server-tts3vkcpgq-uc.a.run.app/?env=Docker_macOS&test=TestMultiNode/serial/ProfileList))|
|[Docker_macOS](https://gopogh-server-tts3vkcpgq-uc.a.run.app/?env=Docker_macOS)|TestMultiNode/serial/RestartKeepsNodes ([gopogh](https://storage.googleapis.com/minikube-builds/logs/17314/31280/Docker_macOS.html#fail_TestMultiNode/serial/RestartKeepsNodes))|5.08 ([chart](https://gopogh-server-tts3vkcpgq-uc.a.run.app/?env=Docker_macOS&test=TestMultiNode/serial/RestartKeepsNodes))|
|[Docker_macOS](https://gopogh-server-tts3vkcpgq-uc.a.run.app/?env=Docker_macOS)|TestMultiNode/serial/RestartMultiNode ([gopogh](https://storage.googleapis.com/minikube-builds/logs/17314/31280/Docker_macOS.html#fail_TestMultiNode/serial/RestartMultiNode))|5.08 ([chart](https://gopogh-server-tts3vkcpgq-uc.a.run.app/?env=Docker_macOS&test=TestMultiNode/serial/RestartMultiNode))|
|[Docker_macOS](https://gopogh-server-tts3vkcpgq-uc.a.run.app/?env=Docker_macOS)|TestMultiNode/serial/StartAfterStop ([gopogh](https://storage.googleapis.com/minikube-builds/logs/17314/31280/Docker_macOS.html#fail_TestMultiNode/serial/StartAfterStop))|5.08 ([chart](https://gopogh-server-tts3vkcpgq-uc.a.run.app/?env=Docker_macOS&test=TestMultiNode/serial/StartAfterStop))|
|[Docker_macOS](https://gopogh-server-tts3vkcpgq-uc.a.run.app/?env=Docker_macOS)|TestMultiNode/serial/StopMultiNode ([gopogh](https://storage.googleapis.com/minikube-builds/logs/17314/31280/Docker_macOS.html#fail_TestMultiNode/serial/StopMultiNode))|5.08 ([chart](https://gopogh-server-tts3vkcpgq-uc.a.run.app/?env=Docker_macOS&test=TestMultiNode/serial/StopMultiNode))|
|[Docker_macOS](https://gopogh-server-tts3vkcpgq-uc.a.run.app/?env=Docker_macOS)|TestMultiNode/serial/StopNode ([gopogh](https://storage.googleapis.com/minikube-builds/logs/17314/31280/Docker_macOS.html#fail_TestMultiNode/serial/StopNode))|5.08 ([chart](https://gopogh-server-tts3vkcpgq-uc.a.run.app/?env=Docker_macOS&test=TestMultiNode/serial/StopNode))|
|[Docker_macOS](https://gopogh-server-tts3vkcpgq-uc.a.run.app/?env=Docker_macOS)|TestOffline ([gopogh](https://storage.googleapis.com/minikube-builds/logs/17314/31280/Docker_macOS.html#fail_TestOffline))|5.08 ([chart](https://gopogh-server-tts3vkcpgq-uc.a.run.app/?env=Docker_macOS&test=TestOffline))|
|[Docker_macOS](https://gopogh-server-tts3vkcpgq-uc.a.run.app/?env=Docker_macOS)|TestScheduledStopUnix ([gopogh](https://storage.googleapis.com/minikube-builds/logs/17314/31280/Docker_macOS.html#fail_TestScheduledStopUnix))|5.08 ([chart](https://gopogh-server-tts3vkcpgq-uc.a.run.app/?env=Docker_macOS&test=TestScheduledStopUnix))|
|[Docker_macOS](https://gopogh-server-tts3vkcpgq-uc.a.run.app/?env=Docker_macOS)|TestSkaffold ([gopogh](https://storage.googleapis.com/minikube-builds/logs/17314/31280/Docker_macOS.html#fail_TestSkaffold))|5.08 ([chart](https://gopogh-server-tts3vkcpgq-uc.a.run.app/?env=Docker_macOS&test=TestSkaffold))|
|[QEMU_macOS](https://gopogh-server-tts3vkcpgq-uc.a.run.app/?env=QEMU_macOS)|TestFunctional/parallel/TunnelCmd/serial/RunSecondTunnel ([gopogh](https://storage.googleapis.com/minikube-builds/logs/17314/31280/QEMU_macOS.html#fail_TestFunctional/parallel/TunnelCmd/serial/RunSecondTunnel))|48.03 ([chart](https://gopogh-server-tts3vkcpgq-uc.a.run.app/?env=QEMU_macOS&test=TestFunctional/parallel/TunnelCmd/serial/RunSecondTunnel))|


To see the flake rates of all tests by environment, click [here](https://minikube.sigs.k8s.io/docs/contrib/test_flakes/).
k8s-ci-robot: @spowelljr: The following test **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-minikube-build | 8569da13ca04003646dc49d23722d84408f35a61 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/minikube/17314/pull-minikube-build/1709692040603766784) | true | `/test pull-minikube-build`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes&repo=minikube&pr=17314). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Aspowelljr). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes/minikube/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
minikube-pr-bot: **kvm2 driver with docker runtime**
```
+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 17314) |
+----------------+----------+---------------------+
| minikube start | 50.9s    | 50.8s               |
| enable ingress | 26.8s    | 27.5s               |
+----------------+----------+---------------------+
```

<details>

Times for minikube ingress: 26.6s 25.7s 26.7s 28.1s 27.2s 
Times for minikube (PR 17314) ingress: 23.7s 29.1s 28.7s 28.1s 28.2s 

Times for minikube start: 51.2s 49.0s 48.7s 54.5s 51.1s 
Times for minikube (PR 17314) start: 49.8s 51.9s 50.7s 50.4s 51.0s 

</details>

**docker driver with docker runtime**
```
+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 17314) |
+----------------+----------+---------------------+
| minikube start | 22.8s    | 23.4s               |
| enable ingress | 21.1s    | 20.9s               |
+----------------+----------+---------------------+
```

<details>

Times for minikube start: 24.4s 22.0s 21.6s 21.4s 24.5s 
Times for minikube (PR 17314) start: 23.1s 21.6s 25.2s 25.0s 22.1s 

Times for minikube ingress: 20.8s 20.8s 21.3s 20.8s 21.9s 
Times for minikube (PR 17314) ingress: 20.8s 20.9s 20.8s 20.9s 21.3s 

</details>

**docker driver with containerd runtime**
```
+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 17314) |
+----------------+----------+---------------------+
| minikube start | 21.9s    | 23.0s               |
| enable ingress | 31.3s    | 31.3s               |
+----------------+----------+---------------------+
```

<details>

Times for minikube start: 21.3s 21.0s 23.0s 23.5s 20.6s 
Times for minikube (PR 17314) start: 23.5s 23.6s 20.5s 23.0s 24.6s 

Times for minikube ingress: 32.3s 30.4s 31.4s 31.3s 31.3s 
Times for minikube (PR 17314) ingress: 31.3s 31.4s 31.3s 31.3s 31.3s 

</details>
minikube-pr-bot: **kvm2 driver with docker runtime**
```
+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 17314) |
+----------------+----------+---------------------+
| minikube start | 51.3s    | 50.0s               |
| enable ingress | 28.9s    | 28.4s               |
+----------------+----------+---------------------+
```

<details>

Times for minikube start: 53.7s 48.8s 52.0s 51.4s 50.6s 
Times for minikube (PR 17314) start: 50.3s 50.5s 50.1s 49.9s 49.1s 

Times for minikube ingress: 30.1s 28.2s 28.1s 29.2s 28.6s 
Times for minikube (PR 17314) ingress: 27.2s 29.7s 28.1s 28.2s 28.6s 

</details>

**docker driver with docker runtime**
```
+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 17314) |
+----------------+----------+---------------------+
| minikube start | 23.8s    | 24.1s               |
| enable ingress | 21.9s    | 21.0s               |
+----------------+----------+---------------------+
```

<details>

Times for minikube (PR 17314) ingress: 21.8s 20.8s 20.8s 20.8s 20.8s 
Times for minikube ingress: 22.8s 21.8s 22.3s 20.4s 22.3s 

Times for minikube start: 23.9s 22.4s 22.2s 24.9s 25.8s 
Times for minikube (PR 17314) start: 25.5s 24.5s 28.1s 21.4s 21.3s 

</details>

**docker driver with containerd runtime**
```
+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 17314) |
+----------------+----------+---------------------+
| minikube start | 22.2s    | 22.1s               |
| enable ingress | 34.5s    | 31.5s               |
+----------------+----------+---------------------+
```

<details>

Times for minikube start: 23.6s 23.7s 19.9s 23.7s 20.1s 
Times for minikube (PR 17314) start: 22.3s 20.6s 23.8s 20.6s 22.9s 

Times for minikube ingress: 31.3s 31.3s 32.3s 46.3s 31.3s 
Times for minikube (PR 17314) ingress: 31.3s 31.3s 31.3s 31.3s 32.3s 

</details>
minikube-pr-bot: These are the flake rates of all failed tests.
|Environment|Failed Tests|Flake Rate (%)|
|---|---|---|
|[Hyper-V_Windows](https://gopogh-server-tts3vkcpgq-uc.a.run.app/?env=Hyper-V_Windows)|TestCertExpiration ([gopogh](https://storage.googleapis.com/minikube-builds/logs/17314/31312/Hyper-V_Windows.html#fail_TestCertExpiration))|0.00 ([chart](https://gopogh-server-tts3vkcpgq-uc.a.run.app/?env=Hyper-V_Windows&test=TestCertExpiration))|
|[Hyper-V_Windows](https://gopogh-server-tts3vkcpgq-uc.a.run.app/?env=Hyper-V_Windows)|TestDockerFlags ([gopogh](https://storage.googleapis.com/minikube-builds/logs/17314/31312/Hyper-V_Windows.html#fail_TestDockerFlags))|0.00 ([chart](https://gopogh-server-tts3vkcpgq-uc.a.run.app/?env=Hyper-V_Windows&test=TestDockerFlags))|
|[Hyper-V_Windows](https://gopogh-server-tts3vkcpgq-uc.a.run.app/?env=Hyper-V_Windows)|TestMinikubeProfile ([gopogh](https://storage.googleapis.com/minikube-builds/logs/17314/31312/Hyper-V_Windows.html#fail_TestMinikubeProfile))|0.00 ([chart](https://gopogh-server-tts3vkcpgq-uc.a.run.app/?env=Hyper-V_Windows&test=TestMinikubeProfile))|
|[Hyper-V_Windows](https://gopogh-server-tts3vkcpgq-uc.a.run.app/?env=Hyper-V_Windows)|TestMultiNode/serial/RestartMultiNode ([gopogh](https://storage.googleapis.com/minikube-builds/logs/17314/31312/Hyper-V_Windows.html#fail_TestMultiNode/serial/RestartMultiNode))|0.00 ([chart](https://gopogh-server-tts3vkcpgq-uc.a.run.app/?env=Hyper-V_Windows&test=TestMultiNode/serial/RestartMultiNode))|
|[Hyper-V_Windows](https://gopogh-server-tts3vkcpgq-uc.a.run.app/?env=Hyper-V_Windows)|TestNetworkPlugins/group/custom-flannel/Start ([gopogh](https://storage.googleapis.com/minikube-builds/logs/17314/31312/Hyper-V_Windows.html#fail_TestNetworkPlugins/group/custom-flannel/Start))|0.00 ([chart](https://gopogh-server-tts3vkcpgq-uc.a.run.app/?env=Hyper-V_Windows&test=TestNetworkPlugins/group/custom-flannel/Start))|
|[Hyper-V_Windows](https://gopogh-server-tts3vkcpgq-uc.a.run.app/?env=Hyper-V_Windows)|TestNetworkPlugins/group/flannel/Start ([gopogh](https://storage.googleapis.com/minikube-builds/logs/17314/31312/Hyper-V_Windows.html#fail_TestNetworkPlugins/group/flannel/Start))|0.00 ([chart](https://gopogh-server-tts3vkcpgq-uc.a.run.app/?env=Hyper-V_Windows&test=TestNetworkPlugins/group/flannel/Start))|
|[Hyper-V_Windows](https://gopogh-server-tts3vkcpgq-uc.a.run.app/?env=Hyper-V_Windows)|TestNetworkPlugins/group/auto/Start ([gopogh](https://storage.googleapis.com/minikube-builds/logs/17314/31312/Hyper-V_Windows.html#fail_TestNetworkPlugins/group/auto/Start))|4.35 ([chart](https://gopogh-server-tts3vkcpgq-uc.a.run.app/?env=Hyper-V_Windows&test=TestNetworkPlugins/group/auto/Start))|
|[Hyper-V_Windows](https://gopogh-server-tts3vkcpgq-uc.a.run.app/?env=Hyper-V_Windows)|TestNoKubernetes/serial/StartWithK8s ([gopogh](https://storage.googleapis.com/minikube-builds/logs/17314/31312/Hyper-V_Windows.html#fail_TestNoKubernetes/serial/StartWithK8s))|4.35 ([chart](https://gopogh-server-tts3vkcpgq-uc.a.run.app/?env=Hyper-V_Windows&test=TestNoKubernetes/serial/StartWithK8s))|
|[KVM_Linux](https://gopogh-server-tts3vkcpgq-uc.a.run.app/?env=KVM_Linux)|TestStartStop/group/newest-cni/serial/Pause ([gopogh](https://storage.googleapis.com/minikube-builds/logs/17314/31312/KVM_Linux.html#fail_TestStartStop/group/newest-cni/serial/Pause))|4.44 ([chart](https://gopogh-server-tts3vkcpgq-uc.a.run.app/?env=KVM_Linux&test=TestStartStop/group/newest-cni/serial/Pause))|
|[KVM_Linux](https://gopogh-server-tts3vkcpgq-uc.a.run.app/?env=KVM_Linux)|TestStartStop/group/newest-cni/serial/SecondStart ([gopogh](https://storage.googleapis.com/minikube-builds/logs/17314/31312/KVM_Linux.html#fail_TestStartStop/group/newest-cni/serial/SecondStart))|4.44 ([chart](https://gopogh-server-tts3vkcpgq-uc.a.run.app/?env=KVM_Linux&test=TestStartStop/group/newest-cni/serial/SecondStart))|
|[KVM_Linux](https://gopogh-server-tts3vkcpgq-uc.a.run.app/?env=KVM_Linux)|TestStartStop/group/newest-cni/serial/VerifyKubernetesImages ([gopogh](https://storage.googleapis.com/minikube-builds/logs/17314/31312/KVM_Linux.html#fail_TestStartStop/group/newest-cni/serial/VerifyKubernetesImages))|4.44 ([chart](https://gopogh-server-tts3vkcpgq-uc.a.run.app/?env=KVM_Linux&test=TestStartStop/group/newest-cni/serial/VerifyKubernetesImages))|
|[Hyperkit_macOS](https://gopogh-server-tts3vkcpgq-uc.a.run.app/?env=Hyperkit_macOS)|TestMultiNode/serial/ValidateNameConflict ([gopogh](https://storage.googleapis.com/minikube-builds/logs/17314/31312/Hyperkit_macOS.html#fail_TestMultiNode/serial/ValidateNameConflict))|14.29 ([chart](https://gopogh-server-tts3vkcpgq-uc.a.run.app/?env=Hyperkit_macOS&test=TestMultiNode/serial/ValidateNameConflict))|
|[Hyper-V_Windows](https://gopogh-server-tts3vkcpgq-uc.a.run.app/?env=Hyper-V_Windows)|TestRunningBinaryUpgrade ([gopogh](https://storage.googleapis.com/minikube-builds/logs/17314/31312/Hyper-V_Windows.html#fail_TestRunningBinaryUpgrade))|39.13 ([chart](https://gopogh-server-tts3vkcpgq-uc.a.run.app/?env=Hyper-V_Windows&test=TestRunningBinaryUpgrade))|


To see the flake rates of all tests by environment, click [here](https://minikube.sigs.k8s.io/docs/contrib/test_flakes/).
k8s-ci-robot: @spowelljr: The following test **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-minikube-build | 07b1fcc62ab5a25da62e79d29489917e0871fd27 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/minikube/17314/pull-minikube-build/1710067764934217728) | true | `/test pull-minikube-build`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes&repo=minikube&pr=17314). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Aspowelljr). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes/minikube/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
minikube-pr-bot: **kvm2 driver with docker runtime**
```
+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 17314) |
+----------------+----------+---------------------+
| minikube start | 49.6s    | 51.3s               |
| enable ingress | 27.9s    | 28.3s               |
+----------------+----------+---------------------+
```

<details>

Times for minikube start: 48.3s 51.5s 51.4s 49.1s 47.9s 
Times for minikube (PR 17314) start: 50.3s 49.4s 50.8s 50.9s 55.2s 

Times for minikube ingress: 29.2s 27.7s 25.7s 28.7s 28.1s 
Times for minikube (PR 17314) ingress: 28.6s 28.7s 27.7s 28.2s 28.2s 

</details>

**docker driver with docker runtime**
```
+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 17314) |
+----------------+----------+---------------------+
| minikube start | 24.5s    | 24.0s               |
| enable ingress | 21.0s    | 21.3s               |
+----------------+----------+---------------------+
```

<details>

Times for minikube ingress: 19.8s 20.8s 20.8s 20.9s 22.8s 
Times for minikube (PR 17314) ingress: 20.8s 20.8s 21.8s 21.3s 21.8s 

Times for minikube start: 22.5s 25.6s 23.7s 25.5s 25.2s 
Times for minikube (PR 17314) start: 24.7s 25.9s 21.2s 23.7s 24.3s 

</details>

**docker driver with containerd runtime**
```
+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 17314) |
+----------------+----------+---------------------+
| minikube start | 21.2s    | 22.2s               |
| enable ingress | 34.7s    | 31.5s               |
+----------------+----------+---------------------+
```

<details>

Times for minikube start: 19.4s 23.4s 19.8s 20.6s 22.9s 
Times for minikube (PR 17314) start: 20.1s 22.7s 23.6s 20.3s 24.1s 

Times for minikube ingress: 31.3s 32.3s 47.3s 31.4s 31.3s 
Times for minikube (PR 17314) ingress: 31.3s 31.3s 31.3s 31.3s 32.3s 

</details>
k8s-ci-robot: PR needs rebase.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
</details>

Patch:
diff --git a/cmd/minikube/cmd/start.go b/cmd/minikube/cmd/start.go
index 11cfeabd4..f92447103 100644
--- a/cmd/minikube/cmd/start.go
+++ b/cmd/minikube/cmd/start.go
@@ -1302,6 +1302,12 @@ func validateFlags(cmd *cobra.Command, drvName string) {
 		}
 	}
 
+	if cmd.Flags().Changed(gpus) {
+		if err := validateGPUs(viper.GetString(gpus), drvName, viper.GetString(containerRuntime)); err != nil {
+			exit.Message(reason.Usage, "{{.err}}", out.V{"err": err})
+		}
+	}
+
 	if driver.IsSSH(drvName) {
 		sshIPAddress := viper.GetString(sshIPAddress)
 		if sshIPAddress == "" {
@@ -1438,6 +1444,20 @@ func validateRuntime(rtime string) error {
 	return nil
 }
 
+// validateGPUs validates that a valid option was given, and if so, can it be used with the given configuration
+func validateGPUs(value, drvName, rtime string) error {
+	if value == "" {
+		return nil
+	}
+	if value != "nvidia" && value != "all" {
+		return errors.Errorf(`The gpus flag must be passed a value of "nvidia" or "all"`)
+	}
+	if drvName == constants.Docker && (rtime == constants.Docker || rtime == constants.DefaultContainerRuntime) {
+		return nil
+	}
+	return errors.Errorf("The gpus flag can only be used with the docker driver and docker container-runtime")
+}
+
 func getContainerRuntime(old *config.ClusterConfig) string {
 	paramRuntime := viper.GetString(containerRuntime)
 
diff --git a/cmd/minikube/cmd/start_flags.go b/cmd/minikube/cmd/start_flags.go
index d7b12c8c9..b1cba66a2 100644
--- a/cmd/minikube/cmd/start_flags.go
+++ b/cmd/minikube/cmd/start_flags.go
@@ -142,6 +142,7 @@ const (
 	socketVMnetPath         = "socket-vmnet-path"
 	staticIP                = "static-ip"
 	autoPauseInterval       = "auto-pause-interval"
+	gpus                    = "gpus"
 )
 
 var (
@@ -204,6 +205,7 @@ func initMinikubeFlags() {
 	startCmd.Flags().Bool(disableMetrics, false, "If set, disables metrics reporting (CPU and memory usage), this can improve CPU usage. Defaults to false.")
 	startCmd.Flags().String(staticIP, "", "Set a static IP for the minikube cluster, the IP must be: private, IPv4, and the last octet must be between 2 and 254, for example 192.168.200.200 (Docker and Podman drivers only)")
 	startCmd.Flags().Duration(autoPauseInterval, time.Minute*1, "Duration of inactivity before the minikube VM is paused (default 1m0s).  To disable, set to 0s")
+	startCmd.Flags().StringP(gpus, "g", "", "Allow pods to use your NVIDIA GPUs. Options include: [all,nvidia] (Docker driver with Docker container-runtime only)")
 }
 
 // initKubernetesFlags inits the commandline flags for Kubernetes related options
@@ -595,6 +597,7 @@ func generateNewConfigFromFlags(cmd *cobra.Command, k8sVersion string, rtime str
 		},
 		MultiNodeRequested: viper.GetInt(nodes) > 1,
 		AutoPauseInterval:  viper.GetDuration(autoPauseInterval),
+		GPUs:               viper.GetString(gpus),
 	}
 	cc.VerifyComponents = interpretWaitFlag(*cmd)
 	if viper.GetBool(createMount) && driver.IsKIC(drvName) {
diff --git a/deploy/addons/assets.go b/deploy/addons/assets.go
index d4cf42ae1..a1c11e795 100644
--- a/deploy/addons/assets.go
+++ b/deploy/addons/assets.go
@@ -166,4 +166,8 @@ var (
 	// Kubeflow assets for kubeflow addon
 	//go:embed kubeflow/*.yaml
 	Kubeflow embed.FS
+
+	// NvidiaDevicePlugin assets for nvidia-device-plugin addon
+	//go:embed nvidia-device-plugin/*.tmpl
+	NvidiaDevicePlugin embed.FS
 )
diff --git a/deploy/addons/nvidia-device-plugin/nvidia-device-plugin.yaml.tmpl b/deploy/addons/nvidia-device-plugin/nvidia-device-plugin.yaml.tmpl
new file mode 100644
index 000000000..c05c586ed
--- /dev/null
+++ b/deploy/addons/nvidia-device-plugin/nvidia-device-plugin.yaml.tmpl
@@ -0,0 +1,56 @@
+# Copyright (c) 2019, NVIDIA CORPORATION.  All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+apiVersion: apps/v1
+kind: DaemonSet
+metadata:
+  name: nvidia-device-plugin-daemonset
+  namespace: kube-system
+spec:
+  selector:
+    matchLabels:
+      name: nvidia-device-plugin-ds
+  updateStrategy:
+    type: RollingUpdate
+  template:
+    metadata:
+      labels:
+        name: nvidia-device-plugin-ds
+    spec:
+      tolerations:
+      - key: nvidia.com/gpu
+        operator: Exists
+        effect: NoSchedule
+      # Mark this pod as a critical add-on; when enabled, the critical add-on
+      # scheduler reserves resources for critical add-on pods so that they can
+      # be rescheduled after a failure.
+      # See https://kubernetes.io/docs/tasks/administer-cluster/guaranteed-scheduling-critical-addon-pods/
+      priorityClassName: "system-node-critical"
+      containers:
+      - image: {{.CustomRegistries.NvidiaDevicePlugin | default .ImageRepository | default .Registries.NvidiaDevicePlugin}}{{.Images.NvidiaDevicePlugin}}
+        name: nvidia-device-plugin-ctr
+        env:
+          - name: FAIL_ON_INIT_ERROR
+            value: "false"
+        securityContext:
+          allowPrivilegeEscalation: false
+          capabilities:
+            drop: ["ALL"]
+        volumeMounts:
+        - name: device-plugin
+          mountPath: /var/lib/kubelet/device-plugins
+      volumes:
+      - name: device-plugin
+        hostPath:
+          path: /var/lib/kubelet/device-plugins
diff --git a/pkg/addons/config.go b/pkg/addons/config.go
index 18764ac40..3e5f96aeb 100644
--- a/pkg/addons/config.go
+++ b/pkg/addons/config.go
@@ -227,4 +227,9 @@ var Addons = []*Addon{
 		set:       SetBool,
 		callbacks: []setFn{EnableOrDisableAddon},
 	},
+	{
+		name:      "nvidia-device-plugin",
+		set:       SetBool,
+		callbacks: []setFn{EnableOrDisableAddon},
+	},
 }
diff --git a/pkg/drivers/kic/kic.go b/pkg/drivers/kic/kic.go
index 3596a9c24..0b2f3c7ae 100644
--- a/pkg/drivers/kic/kic.go
+++ b/pkg/drivers/kic/kic.go
@@ -88,6 +88,7 @@ func (d *Driver) Create() error {
 		ExtraArgs:     append([]string{"--expose", fmt.Sprintf("%d", d.NodeConfig.APIServerPort)}, d.NodeConfig.ExtraArgs...),
 		OCIBinary:     d.NodeConfig.OCIBinary,
 		APIServerPort: d.NodeConfig.APIServerPort,
+		GPUs:          d.NodeConfig.GPUs,
 	}
 
 	networkName := d.NodeConfig.Network
diff --git a/pkg/drivers/kic/oci/oci.go b/pkg/drivers/kic/oci/oci.go
index 29dc26293..986f4a72a 100644
--- a/pkg/drivers/kic/oci/oci.go
+++ b/pkg/drivers/kic/oci/oci.go
@@ -190,6 +190,9 @@ func CreateContainerNode(p CreateParams) error {
 		runArgs = append(runArgs, "--network", p.Network)
 		runArgs = append(runArgs, "--ip", p.IP)
 	}
+	if p.GPUs != "" {
+		runArgs = append(runArgs, "--gpus", "all")
+	}
 
 	memcgSwap := hasMemorySwapCgroup()
 	memcg := HasMemoryCgroup()
diff --git a/pkg/drivers/kic/oci/types.go b/pkg/drivers/kic/oci/types.go
index 0b0efb471..1009df9c7 100644
--- a/pkg/drivers/kic/oci/types.go
+++ b/pkg/drivers/kic/oci/types.go
@@ -58,7 +58,8 @@ type CreateParams struct {
 	ExtraArgs     []string          // a list of any extra option to pass to oci binary during creation time, for example --expose 8080...
 	OCIBinary     string            // docker or podman
 	Network       string            // network name that the container will attach to
-	IP            string            // static IP to assign for th container in the cluster network
+	IP            string            // static IP to assign the container in the cluster network
+	GPUs          string            // add NVIDIA GPU devices to the container
 }
 
 // createOpt is an option for Create
diff --git a/pkg/drivers/kic/types.go b/pkg/drivers/kic/types.go
index a064afcea..cd08a7b81 100644
--- a/pkg/drivers/kic/types.go
+++ b/pkg/drivers/kic/types.go
@@ -69,4 +69,5 @@ type Config struct {
 	StaticIP          string            // static IP for the kic cluster
 	ExtraArgs         []string          // a list of any extra option to pass to oci binary during creation time, for example --expose 8080...
 	ListenAddress     string            // IP Address to listen to
+	GPUs              string            // add NVIDIA GPU devices to the container
 }
diff --git a/pkg/minikube/assets/addons.go b/pkg/minikube/assets/addons.go
index 1639ba165..f926d3b07 100644
--- a/pkg/minikube/assets/addons.go
+++ b/pkg/minikube/assets/addons.go
@@ -93,6 +93,11 @@ func (a *Addon) IsEnabledOrDefault(cc *config.ClusterConfig) bool {
 	return a.enabled
 }
 
+// EnableByDefault will enable the addon by default on cluster start
+func (a *Addon) EnableByDefault() {
+	a.enabled = true
+}
+
 // Addons is the list of addons
 // TODO: Make dynamically loadable: move this data to a .yaml file within each addon directory
 var Addons = map[string]*Addon{
@@ -774,6 +779,14 @@ var Addons = map[string]*Addon{
 		MustBinAsset(addons.Kubeflow, "kubeflow/kubeflow.yaml", vmpath.GuestAddonsDir, "kubeflow.yaml", "0640"),
 	}, false, "kubeflow", "3rd party", "", "", nil, nil,
 	),
+	"nvidia-device-plugin": NewAddon([]*BinAsset{
+		MustBinAsset(addons.NvidiaDevicePlugin, "nvidia-device-plugin/nvidia-device-plugin.yaml.tmpl", vmpath.GuestAddonsDir, "nvidia-device-plugin.yaml", "0640"),
+	}, false, "nvidia-device-plugin", "3rd party (NVIDIA)", "", "",
+		map[string]string{
+			"NvidiaDevicePlugin": "nvidia/k8s-device-plugin:v0.14.1@sha256:15c4280d13a61df703b12d1fd1b5b5eec4658157db3cb4b851d3259502310136",
+		}, map[string]string{
+			"NvidiaDevicePlugin": "nvcr.io",
+		}),
 }
 
 // parseMapString creates a map based on `str` which is encoded as <key1>=<value1>,<key2>=<value2>,...
diff --git a/pkg/minikube/config/types.go b/pkg/minikube/config/types.go
index e38b4cf40..245f5c10e 100644
--- a/pkg/minikube/config/types.go
+++ b/pkg/minikube/config/types.go
@@ -108,6 +108,7 @@ type ClusterConfig struct {
 	SSHAuthSock             string
 	SSHAgentPID             int
 	AutoPauseInterval       time.Duration // Specifies interval of time to wait before checking if cluster should be paused
+	GPUs                    string
 }
 
 // KubernetesConfig contains the parameters used to configure the VM Kubernetes.
diff --git a/pkg/minikube/cruntime/cruntime.go b/pkg/minikube/cruntime/cruntime.go
index 85488a7ee..d96e230a2 100644
--- a/pkg/minikube/cruntime/cruntime.go
+++ b/pkg/minikube/cruntime/cruntime.go
@@ -50,7 +50,7 @@ func (cs ContainerState) String() string {
 
 // ValidRuntimes lists the supported container runtimes
 func ValidRuntimes() []string {
-	return []string{"docker", "nvidia-docker", "cri-o", "containerd"}
+	return []string{"docker", "cri-o", "containerd"}
 }
 
 // CommandRunner is the subset of command.Runner this package consumes
@@ -155,6 +155,8 @@ type Config struct {
 	KubernetesVersion semver.Version
 	// InsecureRegistry list of insecure registries
 	InsecureRegistry []string
+	// GPUs add GPU devices to the container
+	GPUs bool
 }
 
 // ListContainersOptions are the options to use for listing containers
@@ -210,7 +212,7 @@ func New(c Config) (Manager, error) {
 	sm := sysinit.New(c.Runner)
 
 	switch c.Type {
-	case "", "docker", "nvidia-docker":
+	case "", "docker":
 		sp := c.Socket
 		cs := ""
 		// There is no more dockershim socket, in Kubernetes version 1.24 and beyond
@@ -219,7 +221,6 @@ func New(c Config) (Manager, error) {
 			cs = "cri-docker.socket"
 		}
 		return &Docker{
-			Type:              c.Type,
 			Socket:            sp,
 			Runner:            c.Runner,
 			NetworkPlugin:     c.NetworkPlugin,
@@ -228,6 +229,7 @@ func New(c Config) (Manager, error) {
 			Init:              sm,
 			UseCRI:            (sp != ""), // !dockershim
 			CRIService:        cs,
+			GPUs:              c.GPUs,
 		}, nil
 	case "crio", "cri-o":
 		return &CRIO{
diff --git a/pkg/minikube/cruntime/docker.go b/pkg/minikube/cruntime/docker.go
index d083f94a3..a509df9e1 100644
--- a/pkg/minikube/cruntime/docker.go
+++ b/pkg/minikube/cruntime/docker.go
@@ -39,6 +39,7 @@ import (
 	"k8s.io/minikube/pkg/minikube/docker"
 	"k8s.io/minikube/pkg/minikube/download"
 	"k8s.io/minikube/pkg/minikube/image"
+	"k8s.io/minikube/pkg/minikube/out"
 	"k8s.io/minikube/pkg/minikube/style"
 	"k8s.io/minikube/pkg/minikube/sysinit"
 )
@@ -67,7 +68,6 @@ func (e *ErrISOFeature) Error() string {
 
 // Docker contains Docker runtime state
 type Docker struct {
-	Type              string
 	Socket            string
 	Runner            CommandRunner
 	NetworkPlugin     string
@@ -76,6 +76,7 @@ type Docker struct {
 	Init              sysinit.Manager
 	UseCRI            bool
 	CRIService        string
+	GPUs              bool
 }
 
 // Name is a human readable name for Docker
@@ -560,7 +561,11 @@ func (r *Docker) configureDocker(driver string) error {
 		},
 		StorageDriver: "overlay2",
 	}
-	if r.Type == "nvidia-docker" {
+	if r.GPUs {
+		if err := r.installNvidiaContainerToolkit(); err != nil {
+			return fmt.Errorf("failed installing the NVIDIA Container Toolkit: %v", err)
+		}
+		assets.Addons["nvidia-device-plugin"].EnableByDefault()
 		daemonConfig.DefaultRuntime = "nvidia"
 		runtimes := &dockerDaemonRuntimes{}
 		runtimes.Nvidia.Path = "/usr/bin/nvidia-container-runtime"
@@ -574,6 +579,26 @@ func (r *Docker) configureDocker(driver string) error {
 	return r.Runner.Copy(ma)
 }
 
+// installNvidiaContainerToolkit installs the NVIDIA Container Toolkit
+// https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html
+func (r *Docker) installNvidiaContainerToolkit() error {
+	out.Styled(style.Warning, "Using GPUs with the Docker driver is experimental, if you experience any issues please report them at: https://github.com/kubernetes/minikube/issues/new/choose")
+	out.Styled(style.Toolkit, "Installing the NVIDIA Container Toolkit...")
+	cmds := []string{
+		"curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg",
+		"curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list",
+		"sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit",
+	}
+
+	for _, cmd := range cmds {
+		c := exec.Command("/bin/bash", "-c", cmd)
+		if _, err := r.Runner.RunCmd(c); err != nil {
+			return err
+		}
+	}
+	return nil
+}
+
 // Preload preloads docker with k8s images:
 // 1. Copy over the preloaded tarball into the VM
 // 2. Extract the preloaded tarball to the correct directory
diff --git a/pkg/minikube/node/start.go b/pkg/minikube/node/start.go
index 9047c8b36..7ea3e02d2 100755
--- a/pkg/minikube/node/start.go
+++ b/pkg/minikube/node/start.go
@@ -395,6 +395,9 @@ func configureRuntimes(runner cruntime.CommandRunner, cc config.ClusterConfig, k
 		KubernetesVersion: kv,
 		InsecureRegistry:  cc.InsecureRegistry,
 	}
+	if cc.GPUs != "" {
+		co.GPUs = true
+	}
 	cr, err := cruntime.New(co)
 	if err != nil {
 		exit.Error(reason.InternalRuntime, "Failed runtime", err)
diff --git a/pkg/minikube/registry/drvs/docker/docker.go b/pkg/minikube/registry/drvs/docker/docker.go
index ed23b569c..5feeefb87 100644
--- a/pkg/minikube/registry/drvs/docker/docker.go
+++ b/pkg/minikube/registry/drvs/docker/docker.go
@@ -90,6 +90,7 @@ func configure(cc config.ClusterConfig, n config.Node) (interface{}, error) {
 		Subnet:            cc.Subnet,
 		StaticIP:          cc.StaticIP,
 		ListenAddress:     cc.ListenAddress,
+		GPUs:              cc.GPUs,
 	}), nil
 }
 
diff --git a/pkg/minikube/style/style.go b/pkg/minikube/style/style.go
index 597c402b8..411ee2162 100644
--- a/pkg/minikube/style/style.go
+++ b/pkg/minikube/style/style.go
@@ -139,6 +139,7 @@ var Config = map[Enum]Options{
 	VerifyingNoLine:  {Prefix: "🤔  ", OmitNewline: true},
 	Verifying:        {Prefix: "🤔  "},
 	CNI:              {Prefix: "🔗  "},
+	Toolkit:          {Prefix: "🛠️   "},
 }
 
 // LowPrefix returns a 7-bit compatible prefix for a style
diff --git a/pkg/minikube/style/style_enum.go b/pkg/minikube/style/style_enum.go
index 19dce3a06..d89ba5eeb 100644
--- a/pkg/minikube/style/style_enum.go
+++ b/pkg/minikube/style/style_enum.go
@@ -105,4 +105,5 @@ const (
 	Warning
 	Workaround
 	CNI
+	Toolkit
 )
diff --git a/site/content/en/docs/handbook/addons/nvidia.md b/site/content/en/docs/tutorials/nvidia.md
similarity index 74%
rename from site/content/en/docs/handbook/addons/nvidia.md
rename to site/content/en/docs/tutorials/nvidia.md
index a28f34be3..c9e236f8e 100644
--- a/site/content/en/docs/handbook/addons/nvidia.md
+++ b/site/content/en/docs/tutorials/nvidia.md
@@ -1,6 +1,6 @@
 ---
-title: "Using the Nvidia Addons"
-linkTitle: "Nvidia"
+title: "Using NVIDIA GPUs with minikube"
+linkTitle: "Using NVIDIA GPUs with minikube"
 weight: 1
 date: 2018-01-02
 ---
@@ -8,17 +8,67 @@ date: 2018-01-02
 ## Prerequisites
 
 - Linux
-- kvm2 driver
 - Latest NVIDIA GPU drivers
+- minikube v1.32.0-beta0 or later (docker driver only)
 
-## Using the KVM2 driver
+## Instructions per driver
 
-When using NVIDIA GPUs with the kvm2 driver, we passthrough spare GPUs on the
+{{% tabs %}}
+{{% tab docker %}}
+## Using the docker driver
+
+- Check if `bpf_jit_harden` is set to `0`
+  ```shell
+  sudo sysctl net.core.bpf_jit_harden
+  ```
+  - If it's not `0` run:
+  ```shell
+  echo "net.core.bpf_jit_harden=0" | sudo tee -a /etc/sysctl.conf
+  sudo sysctl -p
+  ```
+
+- Install the [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html) on your host machine
+
+- Configure Docker:
+  ```shell
+  sudo nvidia-ctk runtime configure --runtime=docker && sudo systemctl restart docker
+  ```
+- Start minikube:
+  ```shell
+  minikube start --driver docker --container-runtime docker --gpus all
+  ```
+{{% /tab %}}
+{{% tab none %}}
+## Using the 'none' driver
+
+NOTE: This approach used to expose GPUs here is different than the approach used
+to expose GPUs with `--driver=kvm`. Please don't mix these instructions.
+
+- Install minikube.
+
+- Install the nvidia driver, nvidia-docker and configure docker with nvidia as
+  the default runtime. See instructions at
+  <https://github.com/NVIDIA/nvidia-docker>
+
+- Start minikube:
+  ```shell
+  minikube start --driver=none --apiserver-ips 127.0.0.1 --apiserver-name localhost
+  ```
+
+- Install NVIDIA's device plugin:
+  ```shell
+  minikube addons enable nvidia-device-plugin
+  ```
+{{% /tab %}}
+{{% tab kvm %}}
+## Using the kvm driver
+
+When using NVIDIA GPUs with the kvm driver, we passthrough spare GPUs on the
 host to the minikube VM. Doing so has a few prerequisites:
 
-- You must install the [kvm2 driver]({{< ref "/docs/drivers/kvm2" >}}) If you already had
+- You must install the [kvm driver]({{< ref "/docs/drivers/kvm2" >}}) If you already had
   this installed make sure that you fetch the latest
-  `docker-machine-driver-kvm2` binary that has GPU support.
+  `docker-machine-driver-kvm` binary that has GPU support.
 
 - Your CPU must support IOMMU. Different vendors have different names for this
   technology. Intel calls it Intel VT-d. AMD calls it AMD-Vi. Your motherboard
@@ -40,9 +90,9 @@ host to the minikube VM. Doing so has a few prerequisites:
   group of these GPUs.
 
 - Once you reboot the system after doing the above, you should be ready to use
-  GPUs with kvm2. Run the following command to start minikube:
+  GPUs with kvm. Run the following command to start minikube:
   ```shell
-  minikube start --driver kvm2 --kvm-gpu
+  minikube start --driver kvm --kvm-gpu
   ```
 
   This command will check if all the above conditions are satisfied and
@@ -68,31 +118,12 @@ host to the minikube VM. Doing so has a few prerequisites:
 See the excellent documentation at
 <https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF>
 
-### Why are so many manual steps required to use GPUs with kvm2 on minikube?
+### Why are so many manual steps required to use GPUs with kvm on minikube?
 
 These steps require elevated privileges which minikube doesn't run with and they
 are disruptive to the host, so we decided to not do them automatically.
-
-## Using the 'none' driver
-
-NOTE: This approach used to expose GPUs here is different than the approach used
-to expose GPUs with `--driver=kvm2`. Please don't mix these instructions.
-
-- Install minikube.
-
-- Install the nvidia driver, nvidia-docker and configure docker with nvidia as
-  the default runtime. See instructions at
-  <https://github.com/NVIDIA/nvidia-docker>
-
-- Start minikube:
-  ```shell
-  minikube start --driver=none --apiserver-ips 127.0.0.1 --apiserver-name localhost
-  ```
-
-- Install NVIDIA's device plugin:
-  ```shell
-  kubectl create -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/master/nvidia-device-plugin.yml
-  ```
+{{% /tab %}}
+{{% /tabs %}}
 
 ## Why does minikube not support NVIDIA GPUs on macOS?
 
@@ -102,7 +133,7 @@ drivers supported by minikube for macOS doesn't support GPU passthrough:
 - [moby/hyperkit#159](https://github.com/moby/hyperkit/issues/159)
 - [VirtualBox docs](https://www.virtualbox.org/manual/ch09.html#pcipassthrough)
 
-Also:
+Also: 
 
 - For quite a while, all Mac hardware (both laptops and desktops) have come with
   Intel or AMD GPUs (and not with NVIDIA GPUs). Recently, Apple added [support


Test patch:
diff --git a/cmd/minikube/cmd/start_test.go b/cmd/minikube/cmd/start_test.go
index 2eed76a7f..a5ad32a6f 100644
--- a/cmd/minikube/cmd/start_test.go
+++ b/cmd/minikube/cmd/start_test.go
@@ -444,7 +444,6 @@ func TestValidateRuntime(t *testing.T) {
 			runtime:  "docker",
 			errorMsg: "",
 		},
-
 		{
 			runtime:  "test",
 			errorMsg: fmt.Sprintf("Invalid Container Runtime: test. Valid runtimes are: %v", cruntime.ValidRuntimes()),
@@ -860,3 +859,32 @@ func TestImageMatchesBinaryVersion(t *testing.T) {
 		}
 	}
 }
+
+func TestValidateGPUs(t *testing.T) {
+	tests := []struct {
+		gpus     string
+		drvName  string
+		runtime  string
+		errorMsg string
+	}{
+		{"", "kvm", "containerd", ""},
+		{"all", "docker", "docker", ""},
+		{"nvidia", "docker", "docker", ""},
+		{"all", "docker", "", ""},
+		{"nvidia", "docker", "", ""},
+		{"all", "kvm", "docker", "The gpus flag can only be used with the docker driver and docker container-runtime"},
+		{"nvidia", "docker", "containerd", "The gpus flag can only be used with the docker driver and docker container-runtime"},
+		{"cat", "docker", "docker", `The gpus flag must be passed a value of "nvidia" or "all"`},
+	}
+
+	for _, tc := range tests {
+		gotError := ""
+		got := validateGPUs(tc.gpus, tc.drvName, tc.runtime)
+		if got != nil {
+			gotError = got.Error()
+		}
+		if gotError != tc.errorMsg {
+			t.Errorf("validateGPUs(%s, %s, %s) = %q; want = %q", tc.gpus, tc.drvName, tc.runtime, got, tc.errorMsg)
+		}
+	}
+}
diff --git a/pkg/minikube/cruntime/cruntime_test.go b/pkg/minikube/cruntime/cruntime_test.go
index ed7e5d6b5..60e068732 100644
--- a/pkg/minikube/cruntime/cruntime_test.go
+++ b/pkg/minikube/cruntime/cruntime_test.go
@@ -40,7 +40,6 @@ func TestName(t *testing.T) {
 	}{
 		{"", "Docker"},
 		{"docker", "Docker"},
-		{"nvidia-docker", "Docker"},
 		{"crio", "CRI-O"},
 		{"cri-o", "CRI-O"},
 		{"containerd", "containerd"},
@@ -125,7 +124,6 @@ func TestCGroupDriver(t *testing.T) {
 		want    string
 	}{
 		{"docker", "cgroupfs"},
-		{"nvidia-docker", "cgroupfs"},
 		{"crio", "cgroupfs"},
 		{"containerd", "cgroupfs"},
 	}
@@ -157,12 +155,6 @@ func TestKubeletOptions(t *testing.T) {
 		{"docker", "1.24.0", map[string]string{
 			"container-runtime-endpoint": "unix:///var/run/cri-dockerd.sock",
 		}},
-		{"nvidia-docker", "1.23.0", map[string]string{
-			"container-runtime": "docker",
-		}},
-		{"nvidia-docker", "1.25.0", map[string]string{
-			"container-runtime-endpoint": "unix:///var/run/cri-dockerd.sock",
-		}},
 		{"crio", "1.25.0", map[string]string{
 			"container-runtime-endpoint": "unix:///var/run/crio/crio.sock",
 		}},
@@ -688,13 +680,6 @@ func TestEnable(t *testing.T) {
 				"crio":          SvcExited,
 				"crio-shutdown": SvcExited,
 			}},
-		{"nvidia-docker", defaultServices,
-			map[string]serviceState{
-				"docker":        SvcRestarted,
-				"containerd":    SvcExited,
-				"crio":          SvcExited,
-				"crio-shutdown": SvcExited,
-			}},
 		{"containerd", defaultServices,
 			map[string]serviceState{
 				"docker":        SvcExited,
@@ -736,7 +721,6 @@ func TestContainerFunctions(t *testing.T) {
 		runtime string
 	}{
 		{"docker"},
-		{"nvidia-docker"},
 		{"crio"},
 		{"containerd"},
 	}
@@ -746,7 +730,7 @@ func TestContainerFunctions(t *testing.T) {
 		t.Run(tc.runtime, func(t *testing.T) {
 			runner := NewFakeRunner(t)
 			prefix := ""
-			if tc.runtime == "docker" || tc.runtime == "nvidia-docker" {
+			if tc.runtime == "docker" {
 				prefix = "k8s_"
 			}
 			runner.containers = map[string]string{
diff --git a/test/integration/addons_test.go b/test/integration/addons_test.go
index 0ebc38bc8..38e219fa9 100644
--- a/test/integration/addons_test.go
+++ b/test/integration/addons_test.go
@@ -99,7 +99,7 @@ func TestAddons(t *testing.T) {
 		// so we override that here to let minikube auto-detect appropriate cgroup driver
 		os.Setenv(constants.MinikubeForceSystemdEnv, "")
 
-		args := append([]string{"start", "-p", profile, "--wait=true", "--memory=4000", "--alsologtostderr", "--addons=registry", "--addons=metrics-server", "--addons=volumesnapshots", "--addons=csi-hostpath-driver", "--addons=gcp-auth", "--addons=cloud-spanner", "--addons=inspektor-gadget", "--addons=storage-provisioner-rancher"}, StartArgs()...)
+		args := append([]string{"start", "-p", profile, "--wait=true", "--memory=4000", "--alsologtostderr", "--addons=registry", "--addons=metrics-server", "--addons=volumesnapshots", "--addons=csi-hostpath-driver", "--addons=gcp-auth", "--addons=cloud-spanner", "--addons=inspektor-gadget", "--addons=storage-provisioner-rancher", "--addons=nvidia-device-plugin"}, StartArgs()...)
 		if !NoneDriver() { // none driver does not support ingress
 			args = append(args, "--addons=ingress", "--addons=ingress-dns")
 		}
@@ -133,6 +133,7 @@ func TestAddons(t *testing.T) {
 			{"Headlamp", validateHeadlampAddon},
 			{"CloudSpanner", validateCloudSpannerAddon},
 			{"LocalPath", validateLocalPathAddon},
+			{"NvidiaDevicePlugin", validateNvidiaDevicePlugin},
 		}
 		for _, tc := range tests {
 			tc := tc
@@ -942,3 +943,15 @@ func validateDisablingAddonOnNonExistingCluster(ctx context.Context, t *testing.
 		t.Fatalf("unexpected error was returned: %v", err)
 	}
 }
+
+// validateNvidiaDevicePlugin tests the nvidia-device-plugin addon by ensuring the pod comes up and the addon disables
+func validateNvidiaDevicePlugin(ctx context.Context, t *testing.T, profile string) {
+	defer PostMortemLogs(t, profile)
+
+	if _, err := PodWait(ctx, t, profile, "kube-system", "name=nvidia-device-plugin-ds", Minutes(6)); err != nil {
+		t.Fatalf("failed waiting for nvidia-device-plugin-ds pod: %v", err)
+	}
+	if rr, err := Run(t, exec.CommandContext(ctx, Target(), "addons", "disable", "nvidia-device-plugin", "-p", profile)); err != nil {
+		t.Errorf("failed to disable nvidia-device-plugin: args %q : %v", rr.Command(), err)
+	}
+}