Repository: lima-vm/lima
Pull request #2530 (2024-08-05 14:12:18)

vz: add support for nested virtualization
This PR adds support for nested virtualization for the VZ driver via the `nestedVirtualization` yaml config.

~It is dependent on upstream PR https://github.com/Code-Hex/vz/pull/159 getting merged.~ VZ has been forked https://github.com/lima-vm/vz/pull/1.

Nested virtualization is currently limited to m3 devices running macOS 15 or newer.

abiosoft: ## Testing Instructions

⚠️ **This only works on macOS 15 (currently beta) with VZ driver and on M3 devices.** 

For those willing to try this out, these are the steps.

### 1. Clone sources in same directory

Clone my fork of VZ and checkout `enable-nested-virtualization` branch.

```
git clone -b enable-nested-virtualization https://github.com/Code-Hex/vz
```

Clone my fork of Lima and checkout `vz-nested-virtualization` branch.

```
git clone -b vz-nested-virtualization github.com/lima-vm/lima
```

### 2. Modify Lima to use local copy of VZ

Edit `go.mod` in Lima and append the following snippet.

```go.mod
replace github.com/Code-Hex/vz/v3 => ../vz
```

### 3. Build Lima

From the root directory of Lima, perform a build.

**_PS: You might need to unlink Brew version if previously installed with Brew._** `brew unlink lima`

```
make && sudo make install
```

### 4. Start Lima with nested virtualization enabled

Use VZ driver and enabled nested virtualization.

The `lima.yaml` config file should contain the following.

```yaml
vmType: vz
nestedVirtualization: true
```

### 5. Validate nested virtualization in Lima VM

Enter the Lima VM with `lima` (or similar command) and validate the existence of `/dev/kvm`.

```
$ ls -lh /dev/kvm
crw-rw---- 1 user kvm 10, 232 Aug 29 09:06 /dev/kvm
```

If the VM is running Ubuntu, you can install `cpu-checker` package and run `kvm-ok` to further ascertain.

```
$ kvm-ok
INFO: /dev/kvm exists
KVM acceleration can be used
```



asauber: I was able to create a Lima VM with KVM support with your testing instructions.

```
$ limactl shell ubuntunested
WARN[0000] treating lima version "bc39aed.m" from "/Users/asauber/.lima/ubuntunested/lima-version" as very latest release
asauber@lima-ubuntunested:/Users/asauber$ kvm-ok
INFO: /dev/kvm exists
KVM acceleration can be used
```

Very cool. Thank you for working on this.
AkihiroSuda: CI is failing https://github.com/lima-vm/lima/actions/runs/10903645853/job/30258186834?pr=2530

```
# github.com/lima-vm/lima/pkg/vz
Error: pkg/vz/vm_darwin.go:256:18: platformConfig.SetNestedVirtualizationEnabled undefined (type *"github.com/Code-Hex/vz/v3".GenericPlatformConfiguration has no field or method SetNestedVirtualizationEnabled)
make[1]: *** [_output/bin/limactl] Error 1
```
abiosoft: > CI is failing https://github.com/lima-vm/lima/actions/runs/10903645853/job/30258186834?pr=2530

Yes, that is because the upstream PR is not yet merged and that is why I made it a draft PR.
AkihiroSuda: https://github.com/Code-Hex/vz/pull/159 remains still unreviewed, maybe the repo should be forked?

abiosoft: > [Code-Hex/vz#159](https://github.com/Code-Hex/vz/pull/159) remains still unreviewed, maybe the repo should be forked?

https://github.com/lima-vm/vz/pull/1

Patch:
diff --git a/go.mod b/go.mod
index 00041419..10b50c24 100644
--- a/go.mod
+++ b/go.mod
@@ -139,3 +139,7 @@ require (
 // We can't just `require` github.com/inetaf/tcpproxy, as gvisor-tap-vsock
 // still imports inet.af/tcpproxy: https://github.com/containers/gvisor-tap-vsock/pull/399
 replace inet.af/tcpproxy => github.com/inetaf/tcpproxy v0.0.0-20240214030015-3ce58045626c
+
+// Nested virtualization support is yet to be merged into VZ https://github.com/Code-Hex/vz/pull/159.
+// We use our (temporary) fork to add the feature.
+replace github.com/Code-Hex/vz/v3 => github.com/lima-vm/vz/v3 v3.0.0-20241008080607-2a22b5e278ee
diff --git a/go.sum b/go.sum
index 1b45234c..cf364951 100644
--- a/go.sum
+++ b/go.sum
@@ -4,8 +4,6 @@ github.com/AlecAivazis/survey/v2 v2.3.7 h1:6I/u8FvytdGsgonrYsVn2t8t4QiRnh6QSTqkk
 github.com/AlecAivazis/survey/v2 v2.3.7/go.mod h1:xUTIdE4KCOIjsBAE1JYsUPoCqYdZ1reCfTwbto0Fduo=
 github.com/Code-Hex/go-infinity-channel v1.0.0 h1:M8BWlfDOxq9or9yvF9+YkceoTkDI1pFAqvnP87Zh0Nw=
 github.com/Code-Hex/go-infinity-channel v1.0.0/go.mod h1:5yUVg/Fqao9dAjcpzoQ33WwfdMWmISOrQloDRn3bsvY=
-github.com/Code-Hex/vz/v3 v3.1.0 h1:rcMIbZwPYwf78yXOhK68DZgYMdzxlrdmpDuM+NnGf1I=
-github.com/Code-Hex/vz/v3 v3.1.0/go.mod h1:xUfvg1VJ5A6ZQNuzQERwXJ7l2ZdTnY6eCy9CIS6/DYQ=
 github.com/Microsoft/go-winio v0.6.2 h1:F2VQgta7ecxGYO8k3ZZz3RS8fVIXVxONVUPlNERoyfY=
 github.com/Microsoft/go-winio v0.6.2/go.mod h1:yd8OoFMLzJbo9gZq8j5qaps8bJ9aShtEA8Ipt1oGCvU=
 github.com/Netflix/go-expect v0.0.0-20220104043353-73e0943537d2 h1:+vx7roKuyA63nhn5WAunQHLTznkw5W8b1Xc0dNjp83s=
@@ -171,6 +169,8 @@ github.com/lima-vm/go-qcow2reader v0.1.2 h1:A9+h9Yg2oWDxmMlE2oJsdd+djaKk+Ge4hawv
 github.com/lima-vm/go-qcow2reader v0.1.2/go.mod h1:e3p29BzLT8hNh4jbLezdFAHU4eBijf0bm5GilStCRKE=
 github.com/lima-vm/sshocker v0.3.4 h1:5rn6vMkfqwZSZiBW+Udo505OIRhPB4xbLUDdEnFgWwI=
 github.com/lima-vm/sshocker v0.3.4/go.mod h1:QT4c7XNmeQTv79h5/8EgiS7U51B9BLenlXV7idCY0tE=
+github.com/lima-vm/vz/v3 v3.0.0-20241008080607-2a22b5e278ee h1:USiLYd9WbmtU1mPM0egUMrz9QVpMkblMyKioC2EsWCA=
+github.com/lima-vm/vz/v3 v3.0.0-20241008080607-2a22b5e278ee/go.mod h1:WqWQuBbT4SbjO4C4GHG9m9HO8j5jecAmMh4eyVSEbEg=
 github.com/linuxkit/virtsock v0.0.0-20220523201153-1a23e78aa7a2 h1:DZMFueDbfz6PNc1GwDRA8+6lBx1TB9UnxDQliCqR73Y=
 github.com/linuxkit/virtsock v0.0.0-20220523201153-1a23e78aa7a2/go.mod h1:SWzULI85WerrFt3u+nIm5F9l7EvxZTKQvd0InF3nmgM=
 github.com/magiconair/properties v1.8.7 h1:IeQXZAiQcpL9mgcAe1Nu6cX9LLw6ExEHKjN0VQdvPDY=
diff --git a/pkg/limayaml/defaults.go b/pkg/limayaml/defaults.go
index 2ba3534b..ae39921d 100644
--- a/pkg/limayaml/defaults.go
+++ b/pkg/limayaml/defaults.go
@@ -735,6 +735,16 @@ func FillDefault(y, d, o *LimaYAML, filePath string) {
 		y.Rosetta.BinFmt = ptr.Of(false)
 	}
 
+	if y.NestedVirtualization == nil {
+		y.NestedVirtualization = d.NestedVirtualization
+	}
+	if o.NestedVirtualization != nil {
+		y.NestedVirtualization = o.NestedVirtualization
+	}
+	if y.NestedVirtualization == nil {
+		y.NestedVirtualization = ptr.Of(false)
+	}
+
 	if y.Plain == nil {
 		y.Plain = d.Plain
 	}
diff --git a/pkg/limayaml/limayaml.go b/pkg/limayaml/limayaml.go
index ee02018e..b9fcb10c 100644
--- a/pkg/limayaml/limayaml.go
+++ b/pkg/limayaml/limayaml.go
@@ -40,11 +40,12 @@ type LimaYAML struct {
 	DNS          []net.IP          `yaml:"dns,omitempty" json:"dns,omitempty"`
 	HostResolver HostResolver      `yaml:"hostResolver,omitempty" json:"hostResolver,omitempty"`
 	// `useHostResolver` was deprecated in Lima v0.8.1, removed in Lima v0.14.0. Use `hostResolver.enabled` instead.
-	PropagateProxyEnv *bool          `yaml:"propagateProxyEnv,omitempty" json:"propagateProxyEnv,omitempty"`
-	CACertificates    CACertificates `yaml:"caCerts,omitempty" json:"caCerts,omitempty"`
-	Rosetta           Rosetta        `yaml:"rosetta,omitempty" json:"rosetta,omitempty"`
-	Plain             *bool          `yaml:"plain,omitempty" json:"plain,omitempty"`
-	TimeZone          *string        `yaml:"timezone,omitempty" json:"timezone,omitempty"`
+	PropagateProxyEnv    *bool          `yaml:"propagateProxyEnv,omitempty" json:"propagateProxyEnv,omitempty"`
+	CACertificates       CACertificates `yaml:"caCerts,omitempty" json:"caCerts,omitempty"`
+	Rosetta              Rosetta        `yaml:"rosetta,omitempty" json:"rosetta,omitempty"`
+	Plain                *bool          `yaml:"plain,omitempty" json:"plain,omitempty"`
+	TimeZone             *string        `yaml:"timezone,omitempty" json:"timezone,omitempty"`
+	NestedVirtualization *bool          `yaml:"nestedVirtualization,omitempty" json:"nestedVirtualization,omitempty"`
 }
 
 type (
diff --git a/pkg/vz/vm_darwin.go b/pkg/vz/vm_darwin.go
index 56faeec9..fb7da0df 100644
--- a/pkg/vz/vm_darwin.go
+++ b/pkg/vz/vm_darwin.go
@@ -15,6 +15,7 @@ import (
 	"syscall"
 
 	"github.com/Code-Hex/vz/v3"
+	"github.com/coreos/go-semver/semver"
 	"github.com/docker/go-units"
 	"github.com/lima-vm/go-qcow2reader"
 	"github.com/lima-vm/go-qcow2reader/image/raw"
@@ -25,6 +26,7 @@ import (
 	"github.com/lima-vm/lima/pkg/nativeimgutil"
 	"github.com/lima-vm/lima/pkg/networks"
 	"github.com/lima-vm/lima/pkg/networks/usernet"
+	"github.com/lima-vm/lima/pkg/osutil"
 	"github.com/lima-vm/lima/pkg/store"
 	"github.com/lima-vm/lima/pkg/store/filenames"
 	"github.com/sirupsen/logrus"
@@ -234,6 +236,27 @@ func attachPlatformConfig(driver *driver.BaseDriver, vmConfig *vz.VirtualMachine
 	if err != nil {
 		return err
 	}
+
+	// nested virt
+	if *driver.Instance.Config.NestedVirtualization {
+		macOSProductVersion, err := osutil.ProductVersion()
+		if err != nil {
+			return fmt.Errorf("failed to get macOS product version: %w", err)
+		}
+
+		if macOSProductVersion.LessThan(*semver.New("15.0.0")) {
+			return errors.New("nested virtualization requires macOS 15 or newer")
+		}
+
+		if !vz.IsNestedVirtualizationSupported() {
+			return errors.New("nested virtualization is not supported on this device")
+		}
+
+		if err := platformConfig.SetNestedVirtualizationEnabled(true); err != nil {
+			return fmt.Errorf("cannot enable nested virtualization: %w", err)
+		}
+	}
+
 	vmConfig.SetPlatformVirtualMachineConfiguration(platformConfig)
 	return nil
 }
diff --git a/pkg/vz/vz_driver_darwin.go b/pkg/vz/vz_driver_darwin.go
index 28fa8560..0668f7e9 100644
--- a/pkg/vz/vz_driver_darwin.go
+++ b/pkg/vz/vz_driver_darwin.go
@@ -40,6 +40,7 @@ var knownYamlProperties = []string{
 	"Mounts",
 	"MountType",
 	"MountInotify",
+	"NestedVirtualization",
 	"Networks",
 	"OS",
 	"Param",


Test patch:
diff --git a/pkg/limayaml/defaults_test.go b/pkg/limayaml/defaults_test.go
index dd40e667..c5588c08 100644
--- a/pkg/limayaml/defaults_test.go
+++ b/pkg/limayaml/defaults_test.go
@@ -106,7 +106,8 @@ func TestFillDefault(t *testing.T) {
 		CACertificates: CACertificates{
 			RemoveDefaults: ptr.Of(false),
 		},
-		Plain: ptr.Of(false),
+		NestedVirtualization: ptr.Of(false),
+		Plain:                ptr.Of(false),
 	}
 
 	defaultPortForward := PortForward{
@@ -293,6 +294,8 @@ func TestFillDefault(t *testing.T) {
 		BinFmt:  ptr.Of(false),
 	}
 
+	expect.NestedVirtualization = ptr.Of(false)
+
 	FillDefault(&y, &LimaYAML{}, &LimaYAML{}, filePath)
 	assert.DeepEqual(t, &y, &expect, opts...)
 
@@ -420,6 +423,7 @@ func TestFillDefault(t *testing.T) {
 			Enabled: ptr.Of(true),
 			BinFmt:  ptr.Of(true),
 		},
+		NestedVirtualization: ptr.Of(true),
 	}
 
 	expect = d
@@ -634,6 +638,7 @@ func TestFillDefault(t *testing.T) {
 			Enabled: ptr.Of(false),
 			BinFmt:  ptr.Of(false),
 		},
+		NestedVirtualization: ptr.Of(false),
 	}
 
 	y = filledDefaults
@@ -690,6 +695,8 @@ func TestFillDefault(t *testing.T) {
 	}
 	expect.Plain = ptr.Of(false)
 
+	expect.NestedVirtualization = ptr.Of(false)
+
 	FillDefault(&y, &d, &o, filePath)
 	assert.DeepEqual(t, &y, &expect, opts...)
 }