Repository: kubernetes/minikube
Pull request #13639 (2022-02-20 09:38:35)

Add the qemu2 driver to the minikube registry
Since the machine drivers are hardcoded in minikube,
drivers need to be added to the registry to be tested.

Add a basic sanity check for the qemu-system binary,
and set up the basic configuration such as cpus/memory.

----

Most likely needs an updated version of the upstream driver...

https://github.com/machine-drivers/docker-machine-driver-qemu

Note: this driver uses [qemu](https://qemu.org) and runs in userspace but _can_ use kvm

There is another driver called kvm2, which uses the [libvirt](https://libvirt.org) library

k8s-ci-robot: [APPROVALNOTIFIER] This PR is **APPROVED**

This pull-request has been approved by: *<a href="https://github.com/kubernetes/minikube/pull/13639#" title="Author self-approved">afbjorklund</a>*

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=kubernetes%2Fminikube).

The pull request process is described [here](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process)

<details >
Needs approval from an approver in each of these files:

- ~~[OWNERS](https://github.com/kubernetes/minikube/blob/master/OWNERS)~~ [afbjorklund]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->
mprimeaux: @afbjorklund How can I help to progress this PR?
afbjorklund: @mprimeaux : first of all, thanks for volunteering!

This particular PR is more about adding "qemu" to the hard-coded list of allowed machine drivers for minikube
Most likely we will end up having to fork a "qemu2" for minikube, like there was a "kvm2" done for libvirt earlier ?

There are two issues going on here, at the same time...

First we need an ISO that we can test with, and second we need an (Open Source) VM driver that can run it.
Since the current ones are behind in their development, we are trying first to update them for amd64 - then arm64.

----

Ideally, we would want a solution (for `aarch64`) that could work for `minikube` as well as for `docker-machine`.

But that is secondary, working for minikube is primary... Docker stopped all development on the project in 2019.

Our current buildroot is based on BIOS and isolinux, and needs to be changed to use UEFI and grub to boot up.

The current drivers are based on VirtualBox or on HyperKit, which are only available for the x86_64 architecture.
klaases: Hi @mprimeaux, were you able to review @afbjorklund's comments?
mprimeaux: @afbjorklund @klaases Thanks for the overall context and related details.  

Tactically, I am unsure where to begin as I am not as familiar with the Minikube code base (yet). Seems focusing on a buildroot that uses UEFI and grub is a logical first foray. Thoughts?
afbjorklund: It is pretty much two separate projects. And it is possible to start with x86_64, and move on to aarch64.

You will find that this PR is mostly about the minikube driver "registry", and doesn't really have any code.

The main reason why it is so big, is that the config part has been forked from libmachine (for some reason)

https://github.com/machine-drivers/docker-machine-driver-qemu/blob/master/qemu.go#L247_L256

----

The docker-machine drivers use the driver GetCreateFlags and SetConfigFromFlags methods instead.

That means that the drivers can be decoupled (using gRPC), and doesn't have to be included/linked...

So most likely we will end up having to fork "qemu" into a "qemu2", and then adopt it to minikube ?

Note that the driver doesn't really _work_ at the moment, it barely compiles and only for amd64 at that.
k8s-ci-robot: @afbjorklund: PR needs rebase.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
</details>
afbjorklund: The driver actually works now, if you upgrade docker-machine and boot2docker and compile it from source.
afbjorklund: I have added flags to the driver, so that it will be possible to boot different architectures (beyond amd64).

There is a truckload of them, but they can be defaulted. Both machine and cpu are mandatory for arm64.

qemu-program
qemu-machine (new)
qemu-cpu (new)
qemu-bios  (new, bool)
qemu-uefi (new, bool)
qemu-firmware (new)

Patch:
diff --git a/cmd/minikube/cmd/docker-env.go b/cmd/minikube/cmd/docker-env.go
index 2a4efb31c..9f4127518 100644
--- a/cmd/minikube/cmd/docker-env.go
+++ b/cmd/minikube/cmd/docker-env.go
@@ -38,6 +38,7 @@ import (
 	"k8s.io/klog/v2"
 
 	"k8s.io/minikube/pkg/drivers/kic/oci"
+	"k8s.io/minikube/pkg/drivers/qemu"
 	"k8s.io/minikube/pkg/minikube/bootstrapper/bsutil/kverify"
 	"k8s.io/minikube/pkg/minikube/command"
 	"k8s.io/minikube/pkg/minikube/constants"
@@ -290,11 +291,13 @@ var dockerEnvCmd = &cobra.Command{
 
 		d := co.CP.Host.Driver
 		port := constants.DockerDaemonPort
-		if driver.NeedsPortForward(driverName) {
+		if driver.NeedsPortForward(driverName) && driver.IsKIC(driverName) {
 			port, err = oci.ForwardedPort(driverName, cname, port)
 			if err != nil {
 				exit.Message(reason.DrvPortForward, "Error getting port binding for '{{.driver_name}} driver: {{.error}}", out.V{"driver_name": driverName, "error": err})
 			}
+		} else if driver.NeedsPortForward(driverName) && driverName == driver.QEMU2 {
+			port = d.(*qemu.Driver).EnginePort
 		}
 
 		hostname, err := d.GetSSHHostname()
diff --git a/cmd/minikube/cmd/service.go b/cmd/minikube/cmd/service.go
index 2f15ca38a..59e4bfee1 100644
--- a/cmd/minikube/cmd/service.go
+++ b/cmd/minikube/cmd/service.go
@@ -140,8 +140,10 @@ You may select another namespace by using 'minikube service {{.service}} -n <nam
 			}
 		}
 
-		if driver.NeedsPortForward(co.Config.Driver) && services != nil {
+		if driver.NeedsPortForward(co.Config.Driver) && driver.IsKIC(co.Config.Driver) && services != nil {
 			startKicServiceTunnel(services, cname, co.Config.Driver)
+		} else if driver.NeedsPortForward(co.Config.Driver) && driver.IsQEMU(co.Config.Driver) && services != nil {
+			startQemuServiceTunnel(services, cname, co.Config.Driver)
 		} else if !serviceURLMode {
 			openURLs(data)
 		}
@@ -214,6 +216,9 @@ func startKicServiceTunnel(services service.URLs, configName, driverName string)
 	<-ctrlC
 }
 
+func startQemuServiceTunnel(services service.URLs, configName, driverName string) {
+}
+
 func mutateURLs(serviceName string, urls []string) ([]string, error) {
 	formattedUrls := make([]string, 0)
 	for _, rawURL := range urls {
diff --git a/cmd/minikube/cmd/tunnel.go b/cmd/minikube/cmd/tunnel.go
index ac0ef218f..161fa2f13 100644
--- a/cmd/minikube/cmd/tunnel.go
+++ b/cmd/minikube/cmd/tunnel.go
@@ -79,8 +79,7 @@ var tunnelCmd = &cobra.Command{
 			cancel()
 		}()
 
-		if driver.NeedsPortForward(co.Config.Driver) {
-
+		if driver.NeedsPortForward(co.Config.Driver) && driver.IsKIC(co.Config.Driver) {
 			port, err := oci.ForwardedPort(co.Config.Driver, cname, 22)
 			if err != nil {
 				exit.Error(reason.DrvPortForward, "error getting ssh port", err)
diff --git a/go.mod b/go.mod
index e047f9081..9a191bccd 100644
--- a/go.mod
+++ b/go.mod
@@ -45,6 +45,7 @@ require (
 	github.com/juju/version v0.0.0-20180108022336-b64dbd566305 // indirect
 	github.com/kballard/go-shellquote v0.0.0-20180428030007-95032a82bc51
 	github.com/klauspost/cpuid v1.2.0
+	github.com/machine-drivers/docker-machine-driver-qemu v0.1.1-0.20220331133007-0324171328f7
 	github.com/machine-drivers/docker-machine-driver-vmware v0.1.5
 	github.com/mattbaird/jsonpatch v0.0.0-20200820163806-098863c1fc24
 	github.com/mattn/go-isatty v0.0.14
diff --git a/go.sum b/go.sum
index 25759b8fc..671092606 100644
--- a/go.sum
+++ b/go.sum
@@ -369,6 +369,7 @@ github.com/docker/distribution v2.7.1-0.20190205005809-0d3efadf0154+incompatible
 github.com/docker/distribution v2.7.1+incompatible/go.mod h1:J2gT2udsDAN96Uj4KfcMRqY0/ypR+oyYUYmja8H+y+w=
 github.com/docker/distribution v2.8.1+incompatible h1:Q50tZOPR6T/hjNsyc9g8/syEs6bk8XXApsHjKukMl68=
 github.com/docker/distribution v2.8.1+incompatible/go.mod h1:J2gT2udsDAN96Uj4KfcMRqY0/ypR+oyYUYmja8H+y+w=
+github.com/docker/docker v0.0.0-20180621001606-093424bec097/go.mod h1:eEKB0N0r5NX/I1kEveEz05bcu8tLC/8azJZsviup8Sk=
 github.com/docker/docker v17.12.0-ce-rc1.0.20181225093023-5ddb1d410a8b+incompatible/go.mod h1:eEKB0N0r5NX/I1kEveEz05bcu8tLC/8azJZsviup8Sk=
 github.com/docker/docker v17.12.0-ce-rc1.0.20190115220918-5ec31380a5d3+incompatible/go.mod h1:eEKB0N0r5NX/I1kEveEz05bcu8tLC/8azJZsviup8Sk=
 github.com/docker/docker v20.10.7+incompatible/go.mod h1:eEKB0N0r5NX/I1kEveEz05bcu8tLC/8azJZsviup8Sk=
@@ -772,6 +773,8 @@ github.com/liggitt/tabwriter v0.0.0-20181228230101-89fcab3d43de/go.mod h1:zAbeS9
 github.com/lithammer/dedent v1.1.0/go.mod h1:jrXYCQtgg0nJiN+StA2KgR7w6CiQNv9Fd/Z9BP0jIOc=
 github.com/lufia/plan9stats v0.0.0-20211012122336-39d0f177ccd0 h1:6E+4a0GO5zZEnZ81pIr0yLvtUWk2if982qA3F3QD6H4=
 github.com/lufia/plan9stats v0.0.0-20211012122336-39d0f177ccd0/go.mod h1:zJYVVT2jmtg6P3p1VtQj7WsuWi/y4VnjVBn7F8KPB3I=
+github.com/machine-drivers/docker-machine-driver-qemu v0.1.1-0.20220331133007-0324171328f7 h1:f9xnae3LZMVUXFJtqy1xuwQfwX+NQUS5LelCLM3RBxg=
+github.com/machine-drivers/docker-machine-driver-qemu v0.1.1-0.20220331133007-0324171328f7/go.mod h1:yhDK3dYTcmZljNMDPXfmVRwSsHx1EoaEL32v7BANaYs=
 github.com/machine-drivers/docker-machine-driver-vmware v0.1.5 h1:51GqJ84u9EBATnn8rWsHNavcuRPlCLnDmvjzZVuliwY=
 github.com/machine-drivers/docker-machine-driver-vmware v0.1.5/go.mod h1:dTnTzUH3uzhMo0ddV1zRjGYWcVhQWwqiHPxz5l+HPd0=
 github.com/machine-drivers/machine v0.7.1-0.20211105063445-78a84df85426 h1:gVDPCmqwvHQ4ox/9svvnkomYJAAiV59smbPdTK4DIm4=
@@ -1034,6 +1037,7 @@ github.com/shirou/gopsutil/v3 v3.22.4 h1:srAQaiX6jX/cYL6q29aE0m8lOskT9CurZ9N61YR
 github.com/shirou/gopsutil/v3 v3.22.4/go.mod h1:D01hZJ4pVHPpCTZ3m3T2+wDF2YAGfd+H4ifUguaQzHM=
 github.com/shurcooL/sanitized_anchor_name v1.0.0/go.mod h1:1NzhyTcUVG4SuEtjjoZeVRXNmyL/1OwPU0+IJeTBvfc=
 github.com/sirupsen/logrus v1.0.4-0.20170822132746-89742aefa4b2/go.mod h1:pMByvHTf9Beacp5x1UXfOR9xyW/9antXMhjMPG0dEzc=
+github.com/sirupsen/logrus v1.0.4/go.mod h1:pMByvHTf9Beacp5x1UXfOR9xyW/9antXMhjMPG0dEzc=
 github.com/sirupsen/logrus v1.0.6/go.mod h1:pMByvHTf9Beacp5x1UXfOR9xyW/9antXMhjMPG0dEzc=
 github.com/sirupsen/logrus v1.2.0/go.mod h1:LxeOpSwHxABJmUn/MG1IvRgCAasNZTLOkJPxbbu5VWo=
 github.com/sirupsen/logrus v1.3.0/go.mod h1:LxeOpSwHxABJmUn/MG1IvRgCAasNZTLOkJPxbbu5VWo=
@@ -1199,6 +1203,7 @@ go.uber.org/zap v1.19.0/go.mod h1:xg/QME4nWcxGxrpdeYfq7UvYrLh66cuVKdrbD1XF/NI=
 go4.org v0.0.0-20180809161055-417644f6feb5/go.mod h1:MkTOUMDaeVYJUOUsaDXIhWPZYa1yOyC1qaOBpL57BhE=
 golang.org/x/build v0.0.0-20190927031335-2835ba2e683f h1:hXVePvSFG7tPGX4Pwk1d10ePFfoTCc0QmISfpKOHsS8=
 golang.org/x/build v0.0.0-20190927031335-2835ba2e683f/go.mod h1:fYw7AShPAhGMdXqA9gRadk/CcMsvLlClpE5oBwnS3dM=
+golang.org/x/crypto v0.0.0-20170704135851-51714a8c4ac1/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
 golang.org/x/crypto v0.0.0-20171113213409-9f005a07e0d3/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
 golang.org/x/crypto v0.0.0-20180904163835-0709b304e793/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
 golang.org/x/crypto v0.0.0-20181009213950-7c1a557ab941/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
@@ -1375,6 +1380,7 @@ golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJ
 golang.org/x/sync v0.0.0-20201207232520-09787c993a3a/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.0.0-20210220032951-036812b2e83c h1:5KslGYwFpkhGh+Q16bwMP3cOontH8FOep7tGV86Y7SQ=
 golang.org/x/sync v0.0.0-20210220032951-036812b2e83c/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
+golang.org/x/sys v0.0.0-20180202135801-37707fdb30a5/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
 golang.org/x/sys v0.0.0-20180823144017-11551d06cbcc/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
 golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
 golang.org/x/sys v0.0.0-20180905080454-ebe1bf3edb33/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
diff --git a/pkg/addons/addons.go b/pkg/addons/addons.go
index 94016ab2e..2b273dd6d 100644
--- a/pkg/addons/addons.go
+++ b/pkg/addons/addons.go
@@ -250,7 +250,7 @@ func addonSpecificChecks(cc *config.ClusterConfig, name string, enable bool, run
 	}
 
 	if name == "registry" {
-		if driver.NeedsPortForward(cc.Driver) {
+		if driver.NeedsPortForward(cc.Driver) && driver.IsKIC(cc.Driver) {
 			port, err := oci.ForwardedPort(cc.Driver, cc.Name, constants.RegistryAddonPort)
 			if err != nil {
 				return false, errors.Wrap(err, "registry port")
diff --git a/pkg/addons/addons_autopause.go b/pkg/addons/addons_autopause.go
index 3b33b7f9b..2e5faf86e 100644
--- a/pkg/addons/addons_autopause.go
+++ b/pkg/addons/addons_autopause.go
@@ -51,7 +51,7 @@ func enableOrDisableAutoPause(cc *config.ClusterConfig, name, val string) error
 	port := co.CP.Port // API server port
 	if enable {        // if enable, calculate the forwarded port
 		port = constants.AutoPauseProxyPort
-		if driver.NeedsPortForward(cc.Driver) {
+		if driver.NeedsPortForward(cc.Driver) && driver.IsKIC(cc.Driver) {
 			port, err = oci.ForwardedPort(cc.Driver, cc.Name, port)
 			if err != nil {
 				klog.ErrorS(err, "failed to get forwarded port for", "auto-pause port", port)
diff --git a/pkg/drivers/qemu/qemu.go b/pkg/drivers/qemu/qemu.go
new file mode 100644
index 000000000..08ceb5441
--- /dev/null
+++ b/pkg/drivers/qemu/qemu.go
@@ -0,0 +1,770 @@
+/*
+Copyright 2018 The Kubernetes Authors All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package qemu
+
+import (
+	"archive/tar"
+	"bytes"
+	"encoding/json"
+	"fmt"
+	"io/ioutil"
+	"math/rand"
+	"net"
+	"os"
+	"os/exec"
+	"path/filepath"
+	"runtime"
+	"strconv"
+	"strings"
+	"syscall"
+	"time"
+
+	"github.com/docker/machine/libmachine/drivers"
+	"github.com/docker/machine/libmachine/log"
+	"github.com/docker/machine/libmachine/mcnutils"
+	"github.com/docker/machine/libmachine/ssh"
+	"github.com/docker/machine/libmachine/state"
+
+	pkgdrivers "k8s.io/minikube/pkg/drivers"
+)
+
+const (
+	isoFilename        = "boot2docker.iso"
+	privateNetworkName = "docker-machines"
+
+	defaultSSHUser = "docker"
+)
+
+type Driver struct {
+	*drivers.BaseDriver
+	*pkgdrivers.CommonDriver
+	EnginePort int
+	FirstQuery bool
+
+	Memory           int
+	DiskSize         int
+	CPU              int
+	Program          string
+	BIOS             bool
+	CPUType          string
+	MachineType      string
+	Firmware         string
+	Display          bool
+	DisplayType      string
+	Nographic        bool
+	VirtioDrives     bool
+	Network          string
+	PrivateNetwork   string
+	Boot2DockerURL   string
+	NetworkInterface string
+	NetworkAddress   string
+	NetworkSocket    string
+	NetworkBridge    string
+	CaCertPath       string
+	PrivateKeyPath   string
+	DiskPath         string
+	CacheMode        string
+	IOMode           string
+	connectionString string
+	//	conn             *libvirt.Connect
+	//	VM               *libvirt.Domain
+	vmLoaded        bool
+	UserDataFile    string
+	CloudConfigRoot string
+	LocalPorts      string
+}
+
+func (d *Driver) GetMachineName() string {
+	return d.MachineName
+}
+
+func (d *Driver) GetSSHHostname() (string, error) {
+	return "localhost", nil
+}
+
+func (d *Driver) GetSSHKeyPath() string {
+	return d.ResolveStorePath("id_rsa")
+}
+
+func (d *Driver) GetSSHPort() (int, error) {
+	if d.SSHPort == 0 {
+		d.SSHPort = 22
+	}
+
+	return d.SSHPort, nil
+}
+
+func (d *Driver) GetSSHUsername() string {
+	if d.SSHUser == "" {
+		d.SSHUser = "docker"
+	}
+
+	return d.SSHUser
+}
+
+func (d *Driver) DriverName() string {
+	return "qemu2"
+}
+
+func (d *Driver) GetURL() (string, error) {
+	log.Debugf("GetURL called")
+	if _, err := os.Stat(d.pidfilePath()); err != nil {
+		return "", nil
+	}
+	ip, err := d.GetIP()
+	if err != nil {
+		log.Warnf("Failed to get IP: %s", err)
+		return "", err
+	}
+	if ip == "" {
+		return "", nil
+	}
+	port := d.GetPort()
+	return fmt.Sprintf("tcp://%s:%d", ip, port), nil
+}
+
+func NewDriver(hostName, storePath string) drivers.Driver {
+	return &Driver{
+		BIOS:           runtime.GOARCH != "arm64",
+		PrivateNetwork: privateNetworkName,
+		BaseDriver: &drivers.BaseDriver{
+			SSHUser:     defaultSSHUser,
+			MachineName: hostName,
+			StorePath:   storePath,
+		},
+	}
+}
+
+func (d *Driver) GetIP() (string, error) {
+	if d.Network == "user" {
+		return "127.0.0.1", nil
+	}
+	return d.NetworkAddress, nil
+}
+
+func (d *Driver) GetPort() int {
+	var port = d.EnginePort
+	if d.FirstQuery {
+		d.FirstQuery = false
+		port = 2376
+	}
+	return port
+}
+
+func checkPid(pid int) error {
+	process, err := os.FindProcess(pid)
+	if err != nil {
+		return err
+	}
+	return process.Signal(syscall.Signal(0))
+}
+
+func (d *Driver) GetState() (state.State, error) {
+
+	if _, err := os.Stat(d.pidfilePath()); err != nil {
+		return state.Stopped, nil
+	}
+	p, err := ioutil.ReadFile(d.pidfilePath())
+	if err != nil {
+		return state.Error, err
+	}
+	pid, err := strconv.Atoi(strings.TrimSpace(string(p)))
+	if err != nil {
+		return state.Error, err
+	}
+	if err := checkPid(pid); err != nil {
+		// No pid, remove pidfile
+		os.Remove(d.pidfilePath())
+		return state.Stopped, nil
+	}
+	ret, err := d.RunQMPCommand("query-status")
+	if err != nil {
+		return state.Error, err
+	}
+	// RunState is one of:
+	// 'debug', 'inmigrate', 'internal-error', 'io-error', 'paused',
+	// 'postmigrate', 'prelaunch', 'finish-migrate', 'restore-vm',
+	// 'running', 'save-vm', 'shutdown', 'suspended', 'watchdog',
+	// 'guest-panicked'
+	switch ret["status"] {
+	case "running":
+		return state.Running, nil
+	case "paused":
+		return state.Paused, nil
+	case "shutdown":
+		return state.Stopped, nil
+	}
+	return state.None, nil
+}
+
+func (d *Driver) PreCreateCheck() error {
+	return nil
+}
+
+func (d *Driver) Create() error {
+	var err error
+	if d.Network == "user" {
+		minPort, maxPort, err := parsePortRange(d.LocalPorts)
+		log.Debugf("port range: %d -> %d", minPort, maxPort)
+		if err != nil {
+			return err
+		}
+		d.SSHPort, err = getAvailableTCPPortFromRange(minPort, maxPort)
+		if err != nil {
+			return err
+		}
+
+		for {
+			d.EnginePort, err = getAvailableTCPPortFromRange(minPort, maxPort)
+			if err != nil {
+				return err
+			}
+			if d.EnginePort == d.SSHPort {
+				// can't have both on same port
+				continue
+			}
+			break
+		}
+	}
+	b2dutils := mcnutils.NewB2dUtils(d.StorePath)
+	if err := b2dutils.CopyIsoToMachineDir(d.Boot2DockerURL, d.MachineName); err != nil {
+		return err
+	}
+
+	log.Infof("Creating SSH key...")
+	if err := ssh.GenerateSSHKey(d.sshKeyPath()); err != nil {
+		return err
+	}
+
+	log.Infof("Creating Disk image...")
+	if err := d.generateDiskImage(d.DiskSize); err != nil {
+		return err
+	}
+
+	if d.UserDataFile != "" {
+		log.Infof("Creating Userdata Disk...")
+		if d.CloudConfigRoot, err = d.generateUserdataDisk(d.UserDataFile); err != nil {
+			return err
+		}
+	}
+
+	log.Infof("Starting QEMU VM...")
+	return d.Start()
+}
+
+func parsePortRange(rawPortRange string) (int, int, error) {
+	if rawPortRange == "" {
+		return 0, 65535, nil
+	}
+
+	portRange := strings.Split(rawPortRange, "-")
+
+	minPort, err := strconv.Atoi(portRange[0])
+	if err != nil {
+		return 0, 0, fmt.Errorf("Invalid port range")
+	}
+	maxPort, err := strconv.Atoi(portRange[1])
+	if err != nil {
+		return 0, 0, fmt.Errorf("Invalid port range")
+	}
+
+	if maxPort < minPort {
+		return 0, 0, fmt.Errorf("Invalid port range")
+	}
+
+	if maxPort-minPort < 2 {
+		return 0, 0, fmt.Errorf("Port range must be minimum 2 ports")
+	}
+
+	return minPort, maxPort, nil
+}
+
+func getRandomPortNumberInRange(min int, max int) int {
+	return rand.Intn(max-min) + min
+}
+
+func getAvailableTCPPortFromRange(minPort int, maxPort int) (int, error) {
+	port := 0
+	for i := 0; i <= 10; i++ {
+		var ln net.Listener
+		var err error
+		if minPort == 0 && maxPort == 65535 {
+			ln, err = net.Listen("tcp4", "127.0.0.1:0")
+			if err != nil {
+				return 0, err
+			}
+		} else {
+			port = getRandomPortNumberInRange(minPort, maxPort)
+			log.Debugf("testing port: %d", port)
+			ln, err = net.Listen("tcp4", fmt.Sprintf("127.0.0.1:%d", port))
+			if err != nil {
+				log.Debugf("port already in use: %d", port)
+				continue
+			}
+		}
+		defer ln.Close()
+		addr := ln.Addr().String()
+		addrParts := strings.SplitN(addr, ":", 2)
+		p, err := strconv.Atoi(addrParts[1])
+		if err != nil {
+			return 0, err
+		}
+		if p != 0 {
+			port = p
+			return port, nil
+		}
+		time.Sleep(1)
+	}
+	return 0, fmt.Errorf("unable to allocate tcp port")
+}
+
+func (d *Driver) Start() error {
+	// fmt.Printf("Init qemu %s\n", i.VM)
+	machineDir := filepath.Join(d.StorePath, "machines", d.GetMachineName())
+
+	var startCmd []string
+
+	if d.MachineType != "" {
+		machineType := d.MachineType
+		if runtime.GOOS == "darwin" {
+			// highmem=off needed, see https://patchwork.kernel.org/project/qemu-devel/patch/20201126215017.41156-9-agraf@csgraf.de/#23800615 for details
+			machineType += ",accel=hvf,highmem=off"
+		}
+		startCmd = append(startCmd,
+			"-M", machineType,
+		)
+	}
+	if d.CPUType != "" {
+		startCmd = append(startCmd,
+			"-cpu", d.CPUType,
+		)
+	}
+
+	if !d.BIOS {
+		if d.Firmware != "" {
+			startCmd = append(startCmd,
+				"-drive", fmt.Sprintf("file=%s,readonly=on,format=raw,if=pflash", d.Firmware))
+		} else {
+			return fmt.Errorf("unknown firmware")
+		}
+	}
+
+	if d.Display {
+		if d.DisplayType != "" {
+			startCmd = append(startCmd,
+				"-display", d.DisplayType,
+			)
+		} else {
+			// Use the default graphic output
+		}
+	} else {
+		if d.Nographic {
+			startCmd = append(startCmd,
+				"-nographic",
+			)
+		} else {
+			startCmd = append(startCmd,
+				"-display", "none",
+			)
+		}
+	}
+
+	startCmd = append(startCmd,
+		"-m", fmt.Sprintf("%d", d.Memory),
+		"-smp", fmt.Sprintf("%d", d.CPU),
+		"-boot", "d")
+	var isoPath = filepath.Join(machineDir, isoFilename)
+	if d.VirtioDrives {
+		startCmd = append(startCmd,
+			"-drive", fmt.Sprintf("file=%s,index=2,media=cdrom,if=virtio", isoPath))
+	} else {
+		startCmd = append(startCmd,
+			"-cdrom", isoPath)
+	}
+	startCmd = append(startCmd,
+		"-qmp", fmt.Sprintf("unix:%s,server,nowait", d.monitorPath()),
+		"-pidfile", d.pidfilePath(),
+	)
+
+	if d.Network == "user" {
+		startCmd = append(startCmd,
+			"-nic", fmt.Sprintf("user,model=virtio,hostfwd=tcp::%d-:22,hostfwd=tcp::%d-:2376,hostname=%s", d.SSHPort, d.EnginePort, d.GetMachineName()),
+		)
+	} else if d.Network == "tap" {
+		startCmd = append(startCmd,
+			"-nic", fmt.Sprintf("tap,model=virtio,ifname=%s,script=no,downscript=no", d.NetworkInterface),
+		)
+	} else if d.Network == "vde" {
+		startCmd = append(startCmd,
+			"-nic", fmt.Sprintf("vde,model=virtio,sock=%s", d.NetworkSocket),
+		)
+	} else if d.Network == "bridge" {
+		startCmd = append(startCmd,
+			"-nic", fmt.Sprintf("bridge,model=virtio,br=%s", d.NetworkBridge),
+		)
+	} else {
+		log.Errorf("Unknown network: %s", d.Network)
+	}
+
+	startCmd = append(startCmd, "-daemonize")
+
+	// other options
+	// "-enable-kvm" if its available
+	if _, err := os.Stat("/dev/kvm"); err == nil {
+		startCmd = append(startCmd, "-enable-kvm")
+	}
+
+	if d.CloudConfigRoot != "" {
+		startCmd = append(startCmd,
+			"-fsdev",
+			fmt.Sprintf("local,security_model=passthrough,readonly,id=fsdev0,path=%s", d.CloudConfigRoot))
+		startCmd = append(startCmd, "-device", "virtio-9p-pci,id=fs0,fsdev=fsdev0,mount_tag=config-2")
+	}
+
+	if d.VirtioDrives {
+		startCmd = append(startCmd,
+			"-drive", fmt.Sprintf("file=%s,index=0,media=disk,if=virtio", d.diskPath()))
+	} else {
+		// last argument is always the name of the disk image
+		startCmd = append(startCmd, d.diskPath())
+	}
+
+	if stdout, stderr, err := cmdOutErr(d.Program, startCmd...); err != nil {
+		fmt.Printf("OUTPUT: %s\n", stdout)
+		fmt.Printf("ERROR: %s\n", stderr)
+		return err
+	}
+	log.Infof("Waiting for VM to start (ssh -p %d docker@localhost)...", d.SSHPort)
+
+	return WaitForTCPWithDelay(fmt.Sprintf("localhost:%d", d.SSHPort), time.Second)
+}
+
+func cmdOutErr(cmdStr string, args ...string) (string, string, error) {
+	cmd := exec.Command(cmdStr, args...)
+	log.Debugf("executing: %v %v", cmdStr, strings.Join(args, " "))
+	var stdout bytes.Buffer
+	var stderr bytes.Buffer
+	cmd.Stdout = &stdout
+	cmd.Stderr = &stderr
+	err := cmd.Run()
+	stderrStr := stderr.String()
+	log.Debugf("STDOUT: %v", stdout.String())
+	log.Debugf("STDERR: %v", stderrStr)
+	if err != nil {
+		if ee, ok := err.(*exec.Error); ok && ee == exec.ErrNotFound {
+			err = fmt.Errorf("mystery error: %s", ee)
+		}
+	} else {
+		// also catch error messages in stderr, even if the return code
+		// looks OK
+		if strings.Contains(stderrStr, "error:") {
+			err = fmt.Errorf("%v %v failed: %v", cmdStr, strings.Join(args, " "), stderrStr)
+		}
+	}
+	return stdout.String(), stderrStr, err
+}
+
+func (d *Driver) Stop() error {
+	// _, err := d.RunQMPCommand("stop")
+	_, err := d.RunQMPCommand("system_powerdown")
+	if err != nil {
+		return err
+	}
+	return nil
+}
+
+func (d *Driver) Remove() error {
+	s, err := d.GetState()
+	if err != nil {
+		return err
+	}
+	if s == state.Running {
+		if err := d.Kill(); err != nil {
+			return err
+		}
+	}
+	if s != state.Stopped {
+		_, err = d.RunQMPCommand("quit")
+		if err != nil {
+			return err
+		}
+	}
+	return nil
+}
+
+func (d *Driver) Restart() error {
+	s, err := d.GetState()
+	if err != nil {
+		return err
+	}
+
+	if s == state.Running {
+		if err := d.Stop(); err != nil {
+			return err
+		}
+	}
+	return d.Start()
+}
+
+func (d *Driver) Kill() error {
+	// _, err := d.RunQMPCommand("quit")
+	_, err := d.RunQMPCommand("system_powerdown")
+	if err != nil {
+		return err
+	}
+	return nil
+}
+
+func (d *Driver) StartDocker() error {
+	return fmt.Errorf("hosts without a driver cannot start docker")
+}
+
+func (d *Driver) StopDocker() error {
+	return fmt.Errorf("hosts without a driver cannot stop docker")
+}
+
+func (d *Driver) GetDockerConfigDir() string {
+	return ""
+}
+
+func (d *Driver) Upgrade() error {
+	return fmt.Errorf("hosts without a driver cannot be upgraded")
+}
+
+func (d *Driver) sshKeyPath() string {
+	machineDir := filepath.Join(d.StorePath, "machines", d.GetMachineName())
+	return filepath.Join(machineDir, "id_rsa")
+}
+
+func (d *Driver) publicSSHKeyPath() string {
+	return d.sshKeyPath() + ".pub"
+}
+
+func (d *Driver) diskPath() string {
+	machineDir := filepath.Join(d.StorePath, "machines", d.GetMachineName())
+	return filepath.Join(machineDir, "disk.qcow2")
+}
+
+func (d *Driver) monitorPath() string {
+	machineDir := filepath.Join(d.StorePath, "machines", d.GetMachineName())
+	return filepath.Join(machineDir, "monitor")
+}
+
+func (d *Driver) pidfilePath() string {
+	machineDir := filepath.Join(d.StorePath, "machines", d.GetMachineName())
+	return filepath.Join(machineDir, "qemu.pid")
+}
+
+// Make a boot2docker VM disk image.
+func (d *Driver) generateDiskImage(size int) error {
+	log.Debugf("Creating %d MB hard disk image...", size)
+
+	magicString := "boot2docker, please format-me"
+
+	buf := new(bytes.Buffer)
+	tw := tar.NewWriter(buf)
+
+	// magicString first so the automount script knows to format the disk
+	file := &tar.Header{Name: magicString, Size: int64(len(magicString))}
+	if err := tw.WriteHeader(file); err != nil {
+		return err
+	}
+	if _, err := tw.Write([]byte(magicString)); err != nil {
+		return err
+	}
+	// .ssh/key.pub => authorized_keys
+	file = &tar.Header{Name: ".ssh", Typeflag: tar.TypeDir, Mode: 0700}
+	if err := tw.WriteHeader(file); err != nil {
+		return err
+	}
+	pubKey, err := ioutil.ReadFile(d.publicSSHKeyPath())
+	if err != nil {
+		return err
+	}
+	file = &tar.Header{Name: ".ssh/authorized_keys", Size: int64(len(pubKey)), Mode: 0644}
+	if err := tw.WriteHeader(file); err != nil {
+		return err
+	}
+	if _, err := tw.Write(pubKey); err != nil {
+		return err
+	}
+	file = &tar.Header{Name: ".ssh/authorized_keys2", Size: int64(len(pubKey)), Mode: 0644}
+	if err := tw.WriteHeader(file); err != nil {
+		return err
+	}
+	if _, err := tw.Write(pubKey); err != nil {
+		return err
+	}
+	if err := tw.Close(); err != nil {
+		return err
+	}
+	rawFile := fmt.Sprintf("%s.raw", d.diskPath())
+	if err := ioutil.WriteFile(rawFile, buf.Bytes(), 0644); err != nil {
+		return nil
+	}
+	if stdout, stderr, err := cmdOutErr("qemu-img", "convert", "-f", "raw", "-O", "qcow2", rawFile, d.diskPath()); err != nil {
+		fmt.Printf("OUTPUT: %s\n", stdout)
+		fmt.Printf("ERROR: %s\n", stderr)
+		return err
+	}
+	if stdout, stderr, err := cmdOutErr("qemu-img", "resize", d.diskPath(), fmt.Sprintf("+%dM", size)); err != nil {
+		fmt.Printf("OUTPUT: %s\n", stdout)
+		fmt.Printf("ERROR: %s\n", stderr)
+		return err
+	}
+	log.Debugf("DONE writing to %s and %s", rawFile, d.diskPath())
+
+	return nil
+}
+
+func (d *Driver) generateUserdataDisk(userdataFile string) (string, error) {
+	// Start with virtio, add ISO & FAT format later
+	// Start with local file, add wget/fetct URL? (or if URL, use datasource..)
+	userdata, err := ioutil.ReadFile(userdataFile)
+	if err != nil {
+		return "", err
+	}
+
+	machineDir := filepath.Join(d.StorePath, "machines", d.GetMachineName())
+	ccRoot := filepath.Join(machineDir, "cloud-config")
+	err = os.MkdirAll(ccRoot, 0755)
+	if err != nil {
+		return "", err
+	}
+
+	userDataDir := filepath.Join(ccRoot, "openstack/latest")
+	err = os.MkdirAll(userDataDir, 0755)
+	if err != nil {
+		return "", err
+	}
+
+	writeFile := filepath.Join(userDataDir, "user_data")
+	if err := ioutil.WriteFile(writeFile, userdata, 0644); err != nil {
+		return "", err
+	}
+
+	return ccRoot, nil
+
+}
+
+func (d *Driver) RunQMPCommand(command string) (map[string]interface{}, error) {
+
+	// connect to monitor
+	conn, err := net.Dial("unix", d.monitorPath())
+	if err != nil {
+		return nil, err
+	}
+	defer conn.Close()
+
+	// initial QMP response
+	var buf [1024]byte
+	nr, err := conn.Read(buf[:])
+	if err != nil {
+		return nil, err
+	}
+	type qmpInitialResponse struct {
+		QMP struct {
+			Version struct {
+				QEMU struct {
+					Micro int `json:"micro"`
+					Minor int `json:"minor"`
+					Major int `json:"major"`
+				} `json:"qemu"`
+				Package string `json:"package"`
+			} `json:"version"`
+			Capabilities []string `json:"capabilities"`
+		} `jason:"QMP"`
+	}
+
+	var initialResponse qmpInitialResponse
+	err = json.Unmarshal(buf[:nr], &initialResponse)
+	if err != nil {
+		return nil, err
+	}
+
+	// run 'qmp_capabilities' to switch to command mode
+	// { "execute": "qmp_capabilities" }
+	type qmpCommand struct {
+		Command string `json:"execute"`
+	}
+	jsonCommand, err := json.Marshal(qmpCommand{Command: "qmp_capabilities"})
+	if err != nil {
+		return nil, err
+	}
+	_, err = conn.Write(jsonCommand)
+	if err != nil {
+		return nil, err
+	}
+	nr, err = conn.Read(buf[:])
+	if err != nil {
+		return nil, err
+	}
+	type qmpResponse struct {
+		Return map[string]interface{} `json:"return"`
+	}
+	var response qmpResponse
+	err = json.Unmarshal(buf[:nr], &response)
+	if err != nil {
+		return nil, err
+	}
+	// expecting empty response
+	if len(response.Return) != 0 {
+		return nil, fmt.Errorf("qmp_capabilities failed: %v", response.Return)
+	}
+
+	// { "execute": command }
+	jsonCommand, err = json.Marshal(qmpCommand{Command: command})
+	if err != nil {
+		return nil, err
+	}
+	_, err = conn.Write(jsonCommand)
+	if err != nil {
+		return nil, err
+	}
+	nr, err = conn.Read(buf[:])
+	if err != nil {
+		return nil, err
+	}
+	err = json.Unmarshal(buf[:nr], &response)
+	if err != nil {
+		return nil, err
+	}
+	if strings.HasPrefix(command, "query-") {
+		return response.Return, nil
+	}
+	// non-query commands should return an empty response
+	if len(response.Return) != 0 {
+		return nil, fmt.Errorf("%s failed: %v", command, response.Return)
+	}
+	return response.Return, nil
+}
+
+func WaitForTCPWithDelay(addr string, duration time.Duration) error {
+	for {
+		conn, err := net.Dial("tcp", addr)
+		if err != nil {
+			continue
+		}
+		defer conn.Close()
+		if _, err = conn.Read(make([]byte, 1)); err != nil {
+			time.Sleep(duration)
+			continue
+		}
+		break
+	}
+	return nil
+}
diff --git a/pkg/minikube/cluster/ip.go b/pkg/minikube/cluster/ip.go
index bb73ca895..313fb7265 100644
--- a/pkg/minikube/cluster/ip.go
+++ b/pkg/minikube/cluster/ip.go
@@ -58,6 +58,10 @@ func HostIP(host *host.Host, clusterName string) (net.IP, error) {
 			return []byte{}, errors.Wrap(err, "Error converting VM/Host IP address to IPv4 address")
 		}
 		return net.IPv4(vmIP[0], vmIP[1], vmIP[2], byte(1)), nil
+	case driver.QEMU2:
+		return net.ParseIP("10.0.2.2"), nil
+	case driver.QEMU:
+		return net.ParseIP("10.0.2.2"), nil
 	case driver.HyperV:
 		v := reflect.ValueOf(host.Driver).Elem()
 		var hypervVirtualSwitch string
@@ -147,6 +151,9 @@ func DriverIP(api libmachine.API, machineName string) (net.IP, error) {
 	if driver.IsKIC(host.DriverName) {
 		ipStr = oci.DefaultBindIPV4
 	}
+	if driver.IsQEMU(host.DriverName) {
+		ipStr = "127.0.0.1"
+	}
 	ip := net.ParseIP(ipStr)
 	if ip == nil {
 		return nil, fmt.Errorf("parsing IP: %s", ipStr)
diff --git a/pkg/minikube/config/types.go b/pkg/minikube/config/types.go
index c1af9ace7..4566aeb27 100644
--- a/pkg/minikube/config/types.go
+++ b/pkg/minikube/config/types.go
@@ -52,11 +52,12 @@ type ClusterConfig struct {
 	HypervVirtualSwitch     string
 	HypervUseExternalSwitch bool
 	HypervExternalAdapter   string
-	KVMNetwork              string   // Only used by the KVM2 driver
-	KVMQemuURI              string   // Only used by the KVM2 driver
-	KVMGPU                  bool     // Only used by the KVM2 driver
-	KVMHidden               bool     // Only used by the KVM2 driver
-	KVMNUMACount            int      // Only used by the KVM2 driver
+	KVMNetwork              string // Only used by the KVM2 driver
+	KVMQemuURI              string // Only used by the KVM2 driver
+	KVMGPU                  bool   // Only used by the KVM2 driver
+	KVMHidden               bool   // Only used by the KVM2 driver
+	KVMNUMACount            int    // Only used by the KVM2 driver
+	APIServerPort           int
 	DockerOpt               []string // Each entry is formatted as KEY=VALUE.
 	DisableDriverMounts     bool     // Only used by virtualbox
 	NFSShare                []string
diff --git a/pkg/minikube/driver/driver.go b/pkg/minikube/driver/driver.go
index 0b2e317c9..cae2cdd08 100644
--- a/pkg/minikube/driver/driver.go
+++ b/pkg/minikube/driver/driver.go
@@ -46,6 +46,10 @@ const (
 	SSH = "ssh"
 	// KVM2 driver
 	KVM2 = "kvm2"
+	// QEMU2 driver
+	QEMU2 = "qemu2"
+	// QEMU driver
+	QEMU = "qemu"
 	// VirtualBox driver
 	VirtualBox = "virtualbox"
 	// HyperKit driver
@@ -156,6 +160,11 @@ func IsKVM(name string) bool {
 	return name == KVM2 || name == AliasKVM
 }
 
+// IsQEMU checks if the driver is a QEMU[2]
+func IsQEMU(name string) bool {
+	return name == QEMU2 || name == QEMU
+}
+
 // IsVM checks if the driver is a VM
 func IsVM(name string) bool {
 	if IsKIC(name) || BareMetal(name) {
@@ -181,6 +190,9 @@ func AllowsPreload(driverName string) bool {
 
 // NeedsPortForward returns true if driver is unable provide direct IP connectivity
 func NeedsPortForward(name string) bool {
+	if IsQEMU(name) {
+		return true
+	}
 	if !IsKIC(name) {
 		return false
 	}
diff --git a/pkg/minikube/driver/driver_darwin.go b/pkg/minikube/driver/driver_darwin.go
index 195f9841c..cfeba75bb 100644
--- a/pkg/minikube/driver/driver_darwin.go
+++ b/pkg/minikube/driver/driver_darwin.go
@@ -27,6 +27,7 @@ var supportedDrivers = func() []string {
 	if runtime.GOARCH == "arm64" {
 		// on darwin/arm64 only docker and ssh are supported yet
 		return []string{
+			QEMU2,
 			Docker,
 			Podman,
 			SSH,
@@ -50,6 +51,7 @@ var supportedDrivers = func() []string {
 		VMwareFusion,
 		HyperKit,
 		VMware,
+		QEMU2,
 		Docker,
 		Podman,
 		SSH,
diff --git a/pkg/minikube/driver/driver_linux.go b/pkg/minikube/driver/driver_linux.go
index a428a9a2b..e12c23d70 100644
--- a/pkg/minikube/driver/driver_linux.go
+++ b/pkg/minikube/driver/driver_linux.go
@@ -25,6 +25,8 @@ var supportedDrivers = []string{
 	VirtualBox,
 	VMwareFusion,
 	KVM2,
+	QEMU2,
+	QEMU,
 	VMware,
 	None,
 	Docker,
diff --git a/pkg/minikube/driver/driver_windows.go b/pkg/minikube/driver/driver_windows.go
index 8352366ae..51defd622 100644
--- a/pkg/minikube/driver/driver_windows.go
+++ b/pkg/minikube/driver/driver_windows.go
@@ -32,6 +32,7 @@ var supportedDrivers = []string{
 	VMwareFusion,
 	HyperV,
 	VMware,
+	QEMU2,
 	Docker,
 	Podman,
 	SSH,
diff --git a/pkg/minikube/driver/endpoint.go b/pkg/minikube/driver/endpoint.go
index 15203db10..aa02afc9a 100644
--- a/pkg/minikube/driver/endpoint.go
+++ b/pkg/minikube/driver/endpoint.go
@@ -28,7 +28,7 @@ import (
 
 // ControlPlaneEndpoint returns the location where callers can reach this cluster
 func ControlPlaneEndpoint(cc *config.ClusterConfig, cp *config.Node, driverName string) (string, net.IP, int, error) {
-	if NeedsPortForward(driverName) {
+	if NeedsPortForward(driverName) && IsKIC(driverName) {
 		port, err := oci.ForwardedPort(cc.Driver, cc.Name, cp.Port)
 		if err != nil {
 			klog.Warningf("failed to get forwarded control plane port %v", err)
@@ -45,6 +45,8 @@ func ControlPlaneEndpoint(cc *config.ClusterConfig, cp *config.Node, driverName
 			hostname = cc.KubernetesConfig.APIServerName
 		}
 		return hostname, ips[0], port, err
+	} else if NeedsPortForward(driverName) && IsQEMU(driverName) {
+		return "localhost", net.IPv4(127, 0, 0, 1), cc.APIServerPort, nil
 	}
 
 	// https://github.com/kubernetes/minikube/issues/3878
diff --git a/pkg/minikube/machine/machine.go b/pkg/minikube/machine/machine.go
index a465694e0..9f96600fa 100644
--- a/pkg/minikube/machine/machine.go
+++ b/pkg/minikube/machine/machine.go
@@ -122,6 +122,9 @@ func saveHost(api libmachine.API, h *host.Host, cfg *config.ClusterConfig, n *co
 	if err != nil {
 		return err
 	}
+	if ip == "127.0.0.1" && driver.IsQEMU(h.Driver.DriverName()) {
+		ip = "10.0.2.15"
+	}
 	n.IP = ip
 	return config.SaveNode(cfg, n)
 }
diff --git a/pkg/minikube/node/start.go b/pkg/minikube/node/start.go
index 5a8bbaa18..b721f0436 100644
--- a/pkg/minikube/node/start.go
+++ b/pkg/minikube/node/start.go
@@ -253,6 +253,15 @@ func handleAPIServer(starter Starter, cr cruntime.Manager, hostIP net.IP) (*kube
 		return nil, bs, err
 	}
 
+	// Tunnel apiserver to guest, if needed
+	if starter.Cfg.APIServerPort != 0 {
+		args := []string{"-f", "-NTL", fmt.Sprintf("%d:localhost:8443", starter.Cfg.APIServerPort)}
+		err := machine.CreateSSHShell(starter.MachineAPI, *starter.Cfg, *starter.Node, args, false)
+		if err != nil {
+			klog.Warningf("apiserver tunnel failed: %v", err)
+		}
+	}
+
 	// Write the kubeconfig to the file system after everything required (like certs) are created by the bootstrapper.
 	if err := kubeconfig.Update(kcs); err != nil {
 		return nil, bs, errors.Wrap(err, "Failed kubeconfig update")
@@ -558,6 +567,14 @@ func startMachine(cfg *config.ClusterConfig, node *config.Node, delOnFail bool)
 		return runner, preExists, m, host, errors.Wrap(err, "Failed to validate network")
 	}
 
+	if driver.IsQEMU(host.Driver.DriverName()) {
+		apiServerPort, err := getPort()
+		if err != nil {
+			return runner, preExists, m, host, errors.Wrap(err, "Failed to find apiserver port")
+		}
+		cfg.APIServerPort = apiServerPort
+	}
+
 	// Bypass proxy for minikube's vm host ip
 	err = proxy.ExcludeIP(ip)
 	if err != nil {
@@ -567,6 +584,21 @@ func startMachine(cfg *config.ClusterConfig, node *config.Node, delOnFail bool)
 	return runner, preExists, m, host, err
 }
 
+// getPort asks the kernel for a free open port that is ready to use
+func getPort() (int, error) {
+	addr, err := net.ResolveTCPAddr("tcp", "localhost:0")
+	if err != nil {
+		panic(err)
+	}
+
+	l, err := net.ListenTCP("tcp", addr)
+	if err != nil {
+		return -1, errors.Errorf("Error accessing port %d", addr.Port)
+	}
+	defer l.Close()
+	return l.Addr().(*net.TCPAddr).Port, nil
+}
+
 // startHostInternal starts a new minikube host using a VM or None
 func startHostInternal(api libmachine.API, cc *config.ClusterConfig, n *config.Node, delOnFail bool) (*host.Host, bool, error) {
 	host, exists, err := machine.StartHost(api, cc, n)
@@ -638,7 +670,7 @@ func validateNetwork(h *host.Host, r command.Runner, imageRepository string) (st
 		}
 	}
 
-	if !driver.BareMetal(h.Driver.DriverName()) && !driver.IsKIC(h.Driver.DriverName()) {
+	if !driver.BareMetal(h.Driver.DriverName()) && !driver.IsKIC(h.Driver.DriverName()) && !driver.IsQEMU(h.Driver.DriverName()) {
 		if err := trySSH(h, ip); err != nil {
 			return ip, err
 		}
diff --git a/pkg/minikube/registry/drvs/init.go b/pkg/minikube/registry/drvs/init.go
index 06b111582..6176d892e 100644
--- a/pkg/minikube/registry/drvs/init.go
+++ b/pkg/minikube/registry/drvs/init.go
@@ -25,6 +25,8 @@ import (
 	_ "k8s.io/minikube/pkg/minikube/registry/drvs/none"
 	_ "k8s.io/minikube/pkg/minikube/registry/drvs/parallels"
 	_ "k8s.io/minikube/pkg/minikube/registry/drvs/podman"
+	_ "k8s.io/minikube/pkg/minikube/registry/drvs/qemu"
+	_ "k8s.io/minikube/pkg/minikube/registry/drvs/qemu2"
 	_ "k8s.io/minikube/pkg/minikube/registry/drvs/ssh"
 	_ "k8s.io/minikube/pkg/minikube/registry/drvs/virtualbox"
 	_ "k8s.io/minikube/pkg/minikube/registry/drvs/vmware"
diff --git a/pkg/minikube/registry/drvs/qemu/doc.go b/pkg/minikube/registry/drvs/qemu/doc.go
new file mode 100644
index 000000000..ea4425ce8
--- /dev/null
+++ b/pkg/minikube/registry/drvs/qemu/doc.go
@@ -0,0 +1,17 @@
+/*
+Copyright 2018 The Kubernetes Authors All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package qemu
diff --git a/pkg/minikube/registry/drvs/qemu/qemu.go b/pkg/minikube/registry/drvs/qemu/qemu.go
new file mode 100644
index 000000000..24f15693e
--- /dev/null
+++ b/pkg/minikube/registry/drvs/qemu/qemu.go
@@ -0,0 +1,75 @@
+/*
+Copyright 2018 The Kubernetes Authors All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package qemu
+
+import (
+	"fmt"
+	"os/exec"
+	"path/filepath"
+
+	"github.com/docker/machine/libmachine/drivers"
+	drvqemu "github.com/machine-drivers/docker-machine-driver-qemu"
+
+	"k8s.io/minikube/pkg/minikube/config"
+	"k8s.io/minikube/pkg/minikube/download"
+	"k8s.io/minikube/pkg/minikube/driver"
+	"k8s.io/minikube/pkg/minikube/localpath"
+	"k8s.io/minikube/pkg/minikube/registry"
+)
+
+const (
+	docURL = "https://minikube.sigs.k8s.io/docs/reference/drivers/qemu/"
+)
+
+func init() {
+	if err := registry.Register(registry.DriverDef{
+		Name:     driver.QEMU,
+		Config:   configure,
+		Status:   status,
+		Default:  true,
+		Priority: registry.Experimental,
+	}); err != nil {
+		panic(fmt.Sprintf("register failed: %v", err))
+	}
+}
+
+func configure(cc config.ClusterConfig, n config.Node) (interface{}, error) {
+	name := config.MachineName(cc, n)
+	return drvqemu.Driver{
+		BaseDriver: &drivers.BaseDriver{
+			MachineName: name,
+			StorePath:   localpath.MiniPath(),
+			SSHUser:     "docker",
+		},
+		Boot2DockerURL: download.LocalISOResource(cc.MinikubeISO),
+		DiskSize:       cc.DiskSize,
+		Memory:         cc.Memory,
+		CPU:            cc.CPUs,
+		EnginePort:     2376,
+		FirstQuery:     true,
+		DiskPath:       filepath.Join(localpath.MiniPath(), "machines", name, fmt.Sprintf("%s.img", name)),
+	}, nil
+}
+
+func status() registry.State {
+	_, err := exec.LookPath("qemu-system-x86_64")
+	if err != nil {
+		return registry.State{Error: err, Fix: "Install qemu-system", Doc: docURL}
+	}
+
+	return registry.State{Installed: true, Healthy: true, Running: true}
+}
diff --git a/pkg/minikube/registry/drvs/qemu2/doc.go b/pkg/minikube/registry/drvs/qemu2/doc.go
new file mode 100644
index 000000000..c05933150
--- /dev/null
+++ b/pkg/minikube/registry/drvs/qemu2/doc.go
@@ -0,0 +1,17 @@
+/*
+Copyright 2018 The Kubernetes Authors All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package qemu2
diff --git a/pkg/minikube/registry/drvs/qemu2/qemu2.go b/pkg/minikube/registry/drvs/qemu2/qemu2.go
new file mode 100644
index 000000000..229f3ba3c
--- /dev/null
+++ b/pkg/minikube/registry/drvs/qemu2/qemu2.go
@@ -0,0 +1,152 @@
+/*
+Copyright 2018 The Kubernetes Authors All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package qemu2
+
+import (
+	"fmt"
+	"os"
+	"os/exec"
+	"path/filepath"
+	"runtime"
+
+	"github.com/docker/machine/libmachine/drivers"
+	"k8s.io/minikube/pkg/drivers/qemu"
+
+	"k8s.io/minikube/pkg/minikube/config"
+	"k8s.io/minikube/pkg/minikube/download"
+	"k8s.io/minikube/pkg/minikube/driver"
+	"k8s.io/minikube/pkg/minikube/localpath"
+	"k8s.io/minikube/pkg/minikube/registry"
+)
+
+const (
+	docURL = "https://minikube.sigs.k8s.io/docs/reference/drivers/qemu2/"
+)
+
+func init() {
+	if err := registry.Register(registry.DriverDef{
+		Name:     driver.QEMU2,
+		Init:     func() drivers.Driver { return qemu.NewDriver("", "") },
+		Config:   configure,
+		Status:   status,
+		Default:  true,
+		Priority: registry.Experimental,
+	}); err != nil {
+		panic(fmt.Sprintf("register failed: %v", err))
+	}
+}
+
+func qemuSystemProgram() (string, error) {
+	arch := runtime.GOARCH
+	switch arch {
+	case "amd64":
+		return "qemu-system-x86_64", nil
+	case "arm64":
+		return "qemu-system-aarch64", nil
+	default:
+		return "", fmt.Errorf("unknown arch: %s", arch)
+	}
+}
+
+func qemuFirmwarePath() (string, error) {
+	arch := runtime.GOARCH
+	switch arch {
+	case "amd64":
+		// on macOS, we assume qemu is installed via homebrew for simplicity
+		if runtime.GOOS == "darwin" {
+			return "/usr/local/Cellar/qemu/6.2.0_1/share/qemu/edk2-x86_64-code.fd", nil
+		}
+		return "/usr/share/OVMF/OVMF_CODE.fd", nil
+	case "arm64":
+		if runtime.GOOS == "darwin" {
+			return "/opt/homebrew/Cellar/qemu/6.2.0_1/share/qemu/edk2-aarch64-code.fd", nil
+		}
+		return "/usr/share/AAVMF/AAVMF_CODE.fd", nil
+	default:
+		return "", fmt.Errorf("unknown arch: %s", arch)
+	}
+}
+
+func configure(cc config.ClusterConfig, n config.Node) (interface{}, error) {
+	name := config.MachineName(cc, n)
+	qemuSystem, err := qemuSystemProgram()
+	if err != nil {
+		return nil, err
+	}
+	var qemuMachine string
+	var qemuCPU string
+	switch runtime.GOARCH {
+	case "amd64":
+		qemuMachine = "" // default
+		qemuCPU = ""     // default
+	case "arm64":
+		qemuMachine = "virt"
+		qemuCPU = "cortex-a72"
+	default:
+		return nil, fmt.Errorf("unknown arch: %s", runtime.GOARCH)
+	}
+	qemuFirmware, err := qemuFirmwarePath()
+	if err != nil {
+		return nil, err
+	}
+	return qemu.Driver{
+		BaseDriver: &drivers.BaseDriver{
+			MachineName: name,
+			StorePath:   localpath.MiniPath(),
+			SSHUser:     "docker",
+		},
+		Boot2DockerURL: download.LocalISOResource(cc.MinikubeISO),
+		DiskSize:       cc.DiskSize,
+		Memory:         cc.Memory,
+		CPU:            cc.CPUs,
+		EnginePort:     2376,
+		FirstQuery:     true,
+		DiskPath:       filepath.Join(localpath.MiniPath(), "machines", name, fmt.Sprintf("%s.img", name)),
+		Program:        qemuSystem,
+		BIOS:           runtime.GOARCH != "arm64",
+		MachineType:    qemuMachine,
+		CPUType:        qemuCPU,
+		Firmware:       qemuFirmware,
+		VirtioDrives:   false,
+		Network:        "user",
+		CacheMode:      "default",
+		IOMode:         "threads",
+	}, nil
+}
+
+func status() registry.State {
+	qemuSystem, err := qemuSystemProgram()
+	if err != nil {
+		return registry.State{Error: err, Doc: docURL}
+	}
+
+	_, err = exec.LookPath(qemuSystem)
+	if err != nil {
+		return registry.State{Error: err, Fix: "Install qemu-system", Doc: docURL}
+	}
+
+	qemuFirmware, err := qemuFirmwarePath()
+	if err != nil {
+		return registry.State{Error: err, Doc: docURL}
+	}
+
+	if _, err := os.Stat(qemuFirmware); err != nil && runtime.GOARCH == "arm64" {
+		return registry.State{Error: err, Fix: "Install uefi firmware", Doc: docURL}
+	}
+
+	return registry.State{Installed: true, Healthy: true, Running: true}
+}
diff --git a/site/content/en/docs/drivers/_index.md b/site/content/en/docs/drivers/_index.md
index daf996e84..6744c03ff 100644
--- a/site/content/en/docs/drivers/_index.md
+++ b/site/content/en/docs/drivers/_index.md
@@ -17,6 +17,7 @@ To do so, we use the [Docker Machine](https://github.com/docker/machine) library
 * [Docker]({{<ref "docker.md">}}) - container-based (preferred)
 * [KVM2]({{<ref "kvm2.md">}}) - VM-based (preferred)
 * [VirtualBox]({{<ref "virtualbox.md">}}) - VM
+* [QEMU]({{<ref "qemu.md">}}) - VM (experimental)
 * [None]({{<ref "none.md">}}) -  bare-metal
 * [Podman]({{<ref "podman.md">}}) - container (experimental)
 * [SSH]({{<ref "ssh.md">}}) - remote ssh
@@ -29,6 +30,7 @@ To do so, we use the [Docker Machine](https://github.com/docker/machine) library
 * [VirtualBox]({{<ref "virtualbox.md">}}) - VM
 * [Parallels]({{<ref "parallels.md">}}) - VM
 * [VMware Fusion]({{<ref "vmware.md">}}) - VM
+* [QEMU]({{<ref "qemu.md">}}) - VM (experimental)
 * [SSH]({{<ref "ssh.md">}}) - remote ssh
 
 ## Windows
@@ -37,4 +39,5 @@ To do so, we use the [Docker Machine](https://github.com/docker/machine) library
 * [Docker]({{<ref "docker.md">}}) - VM + Container (preferred)
 * [VirtualBox]({{<ref "virtualbox.md">}}) - VM
 * [VMware Workstation]({{<ref "vmware.md">}}) - VM
+* [QEMU]({{<ref "qemu.md">}}) - VM (experimental)
 * [SSH]({{<ref "ssh.md">}}) - remote ssh
diff --git a/site/content/en/docs/drivers/qemu.md b/site/content/en/docs/drivers/qemu.md
new file mode 100644
index 000000000..23167b1e5
--- /dev/null
+++ b/site/content/en/docs/drivers/qemu.md
@@ -0,0 +1,22 @@
+---
+title: "qemu"
+weight: 3
+description: >
+  QEMU driver
+aliases:
+    - /docs/reference/drivers/qemu
+---
+
+## Overview
+
+The `qemu` driver users QEMU (system) for VM creation.
+
+<https://www.qemu.org/>
+
+## Issues
+
+* [Full list of open 'qemu' driver issues](https://github.com/kubernetes/minikube/labels/co%2Fqemu-driver)
+
+## Troubleshooting
+
+* Run `minikube start --alsologtostderr -v=4` to debug crashes


Test patch:
diff --git a/pkg/minikube/driver/driver_test.go b/pkg/minikube/driver/driver_test.go
index 021f78c74..0b7806610 100644
--- a/pkg/minikube/driver/driver_test.go
+++ b/pkg/minikube/driver/driver_test.go
@@ -66,6 +66,8 @@ func TestMachineType(t *testing.T) {
 		None:         "bare metal machine",
 		SSH:          "bare metal machine",
 		KVM2:         "VM",
+		QEMU2:        "VM",
+		QEMU:         "VM",
 		VirtualBox:   "VM",
 		HyperKit:     "VM",
 		VMware:       "VM",