Repository: lima-vm/lima (Go)

Issue:
Windows CI began to fail on Oct 21
#2769 passed the CI, but its merge commit and later ones are failing 

https://github.com/lima-vm/lima/actions/runs/11429806278/job/31800191430

```
[…]
time="2024-10-21T01:14:55Z" level=info msg="SSH Local Port: 22"
time="2024-10-21T01:14:55Z" level=info msg="[hostagent] Waiting for the essential requirement 1 of 2: \"ssh\""
time="2024-10-21T01:15:05Z" level=info msg="[hostagent] Waiting for the essential requirement 1 of 2: \"ssh\""
time="2024-10-21T01:15:15Z" level=info msg="[hostagent] Waiting for the essential requirement 1 of 2: \"ssh\""
time="2024-10-21T01:24:43Z" level=fatal msg="did not receive an event with the \"running\" status"
```

Something seems to have changed between https://github.com/actions/runner-images/releases/tag/win22%2F20241006.1 and https://github.com/actions/runner-images/releases/tag/win22%2F20241015.1


Discussion:
In https://github.com/lima-vm/lima/actions/runs/11445753347/job/31843450765?pr=2778 I see:

```
System has not been booted with systemd as init system (PID 1). Can't operate.
```
@pendo324 Do you have any idea what may be causing the Windows tests to fail now?

I can't find anything that seems related in https://github.com/actions/runner-images/commit/fcc4cdb1d095af1317859c4809364538953b3497 or https://github.com/actions/runner-images/commit/09ff567de6908096a96ace47eb3f41079993366d

The errors look like `systemd` is no longer enabled in your distro, but there has been no change to the distro.

I'm at a loss on what might be causing this.
Thanks for pinging me, taking a look now
I'm doing some experiments with Windows support. I managed to replicate this CI attempt in my rebuild workflow. It worked successfully on a default GH runner. Logs are available https://github.com/arixmkii/qcw/actions/runs/13090314041/job/36526224725

The biggest difference in the setup is that I have to use latest preview WSL build from https://github.com/microsoft/WSL/releases
@jandubois I debugged this. There is actually related change in commits you showed. It is Git version bump. It uses OpenSSH from Git distribution.

The script "user session is ready for ssh" hangs indefinitely on Git 2.47 and newer releases. The same is the case for latest msys2 OpenSSH. I downgraded the Git on my system and managed to run WSL2 machine. 

I also managed to run almost all integration tests with this WSL2 machine, when using OpenSSH inside Alpine companion distro in WSL2 (not using any of Windows tools) https://github.com/arixmkii/qcw/actions/runs/13474629971/job/37652601743

Conclusion. Machine didn't break, Windows tooling has some sort of issue/regression, which might or might not be fixed.

I tried to create an isolated reproducer using same script doing `cat script sh | <openssh command from lima>` in parallel to hanging one and was not able to reproduce it outside of Lima.

FWIW, https://github.com/git-for-windows/git/issues/5199 may be relevant. Note that git-for-Windows picked up the fixes, but as far as I know it's not in upstream cygwin/msys2 yet.

https://github.com/actions/runner-images/commit/fcc4cdb1d095af1317859c4809364538953b3497 linked above shows that Git for Windows was updated to 2.47.0.windows.1 which would be an affected version. But right now it shows 2.47.1.windows.2 which should be fixed (so some runs may be succeeding).
@mook-as Thank you! I tested with the updated runner (I'm using server 2025, but this should not really behave differently here) with `git version 2.47.1.windows.2` and it passed tests https://github.com/arixmkii/qcw/actions/runs/13552437877/job/37879648090



Patch:
diff --git a/hack/common.inc.sh b/hack/common.inc.sh
index 1e90ff8e15d..95e429fce0a 100644
--- a/hack/common.inc.sh
+++ b/hack/common.inc.sh
@@ -23,7 +23,7 @@ if [[ ${BASH_VERSINFO:-0} -lt 4 ]]; then
 	exit 1
 fi
 
-: "${LIMA_HOME:=$HOME/.lima}"
+: "${LIMA_HOME:=${HOME_HOST:-$HOME}/.lima}"
 _IPERF3=iperf3
 # iperf3-darwin does some magic on macOS to avoid "No route on host" on macOS 15
 # https://github.com/lima-vm/socket_vmnet/issues/85
diff --git a/pkg/osutil/user.go b/pkg/osutil/user.go
index d1883c59d6e..a0d89e43cca 100644
--- a/pkg/osutil/user.go
+++ b/pkg/osutil/user.go
@@ -144,6 +144,8 @@ func LimaUser(limaVersion string, warn bool) *user.User {
 			home, err := call([]string{"cygpath", limaUser.HomeDir})
 			if err != nil {
 				logrus.Debug(err)
+			} else {
+				home += ".linux"
 			}
 			if home == "" {
 				drive := filepath.VolumeName(limaUser.HomeDir)
@@ -151,6 +153,7 @@ func LimaUser(limaVersion string, warn bool) *user.User {
 				// replace C: with /c
 				prefix := strings.ToLower(fmt.Sprintf("/%c", drive[0]))
 				home = strings.Replace(home, drive, prefix, 1)
+				home += ".linux"
 			}
 			if !regexPath.MatchString(limaUser.HomeDir) {
 				warning := fmt.Sprintf("local home %q is not a valid Linux path (must match %q); using %q home instead",


Test patch:
diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
index a5157e73a98..5542708ab09 100644
--- a/.github/workflows/test.yml
+++ b/.github/workflows/test.yml
@@ -129,12 +129,10 @@ jobs:
     - name: Enable WSL2
       run: |
         wsl --set-default-version 2
-        # Manually install the latest kernel from MSI
-        Invoke-WebRequest -Uri "https://wslstorestorage.blob.core.windows.net/wslblob/wsl_update_x64.msi" -OutFile "wsl_update_x64.msi"
-        $pwd = (pwd).Path
-        Start-Process msiexec.exe -Wait -ArgumentList "/I $pwd\wsl_update_x64.msi /quiet"
+        wsl --shutdown
         wsl --update
         wsl --status
+        wsl --version
         wsl --list --online
     - name: Install WSL2 distro
       timeout-minutes: 1
@@ -160,22 +158,15 @@ jobs:
       run: go test -v ./...
     - name: Make
       run: make
-# FIXME: Windows CI began to fail on Oct 21, 2024.
-# Something seems to have changed between win22/20241006.1 and win22/20241015.1.
-# https://github.com/lima-vm/lima/issues/2775
-#    - name: Smoke test
-#      # Make sure the path is set properly and then run limactl
-#      run: |
-#        $env:Path = 'C:\Program Files\Git\usr\bin;' + $env:Path
-#        Set-ItemProperty -Path 'Registry::HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Session Manager\Environment' -Name PATH -Value $env:Path
-#        .\_output\bin\limactl.exe start template://experimental/wsl2
-#      # TODO: run the full integration tests
-#    - name: Debug
-#      if: always()
-#      run: type C:\Users\runneradmin\.lima\wsl2\ha.stdout.log
-#    - name: Debug
-#      if: always()
-#      run: type C:\Users\runneradmin\.lima\wsl2\ha.stderr.log
+    - name: Integration tests (WSL2, Windows host)
+      run: |
+        $env:Path = "$pwd\_output\bin;" + 'C:\Program Files\Git\usr\bin;' + $env:Path
+        Set-ItemProperty -Path 'Registry::HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Session Manager\Environment' -Name PATH -Value $env:Path
+        $env:MSYS2_ENV_CONV_EXCL='HOME_HOST;HOME_GUEST'
+        $env:HOME_HOST=$(cygpath.exe "$env:USERPROFILE")
+        $env:HOME_GUEST="/mnt$env:HOME_HOST"
+        $env:LIMACTL_CREATE_ARGS='--vm-type=wsl2 --mount-type=wsl2 --containerd=system'
+        bash.exe -c "./hack/test-templates.sh templates/experimental/wsl2.yaml"
 
   qemu:
     name: "Integration tests (QEMU, macOS host)"
diff --git a/hack/test-mount-home.sh b/hack/test-mount-home.sh
index dd8e1afad72..f5cc4ede751 100755
--- a/hack/test-mount-home.sh
+++ b/hack/test-mount-home.sh
@@ -11,14 +11,15 @@ if [ "$#" -ne 1 ]; then
 fi
 
 NAME="$1"
-hometmp="$HOME/lima-test-tmp"
+hometmp="${HOME_HOST:-$HOME}/lima-test-tmp"
+hometmpguest="${HOME_GUEST:-$HOME}/lima-test-tmp"
 INFO "Testing home access (\"$hometmp\")"
 rm -rf "$hometmp"
 mkdir -p "$hometmp"
 defer "rm -rf \"$hometmp\""
 echo "random-content-${RANDOM}" >"$hometmp/random"
 expected="$(cat "$hometmp/random")"
-got="$(limactl shell "$NAME" cat "$hometmp/random")"
+got="$(limactl shell "$NAME" cat "$hometmpguest/random")"
 INFO "$hometmp/random: expected=${expected}, got=${got}"
 if [ "$got" != "$expected" ]; then
 	ERROR "Home directory is not shared?"
diff --git a/hack/test-port-forwarding.pl b/hack/test-port-forwarding.pl
index 3a41ce1e118..cc1fc96c409 100755
--- a/hack/test-port-forwarding.pl
+++ b/hack/test-port-forwarding.pl
@@ -129,7 +129,8 @@
 sleep 5;
 
 # Record current log size, so we can skip prior output
-$ENV{LIMA_HOME} ||= "$ENV{HOME}/.lima";
+$ENV{HOME_HOST} ||= "$ENV{HOME}";
+$ENV{LIMA_HOME} ||= "$ENV{HOME_HOST}/.lima";
 my $ha_log = "$ENV{LIMA_HOME}/$instance/ha.stderr.log";
 my $ha_log_size = -s $ha_log or die;
 
diff --git a/hack/test-templates.sh b/hack/test-templates.sh
index 26e064b513d..71cdc53e7cd 100755
--- a/hack/test-templates.sh
+++ b/hack/test-templates.sh
@@ -1,6 +1,9 @@
 #!/usr/bin/env bash
 set -eu -o pipefail
 
+# will prevent msys2 converting Linux path arguments into Windows paths before passing to limactl
+export MSYS2_ARG_CONV_EXCL='*'
+
 scriptdir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 # shellcheck source=common.inc.sh
 source "${scriptdir}/common.inc.sh"
@@ -12,9 +15,19 @@ fi
 
 FILE="$1"
 NAME="$(basename -s .yaml "$FILE")"
+OS_HOST="$(uname -o)"
+
+# On Windows $HOME of the bash runner, %USERPROFILE% of the host machine and mpunting point in the guest machine
+# are all different folders. This will handle path differences, when values are expilictly set.
+HOME_HOST=${HOME_HOST:-$HOME}
+HOME_GUEST=${HOME_GUEST:-$HOME}
+FILE_HOST=$FILE
+if [ "${OS_HOST}" = "Msys" ]; then
+	FILE_HOST="$(cygpath -w "$FILE")"
+fi
 
-INFO "Validating \"$FILE\""
-limactl validate "$FILE"
+INFO "Validating \"$FILE_HOST\""
+limactl validate "$FILE_HOST"
 
 # --cpus=1 is needed for running vz on GHA: https://github.com/lima-vm/lima/pull/1511#issuecomment-1574937888
 LIMACTL_CREATE=(limactl --tty=false create --cpus=1 --memory=1)
@@ -22,6 +35,7 @@ LIMACTL_CREATE=(limactl --tty=false create --cpus=1 --memory=1)
 CONTAINER_ENGINE="nerdctl"
 
 declare -A CHECKS=(
+	["proxy-settings"]="1"
 	["systemd"]="1"
 	["systemd-strict"]="1"
 	["mount-home"]="1"
@@ -69,6 +83,13 @@ case "$NAME" in
 "docker")
 	CONTAINER_ENGINE="docker"
 	;;
+"wsl2")
+	# TODO https://github.com/lima-vm/lima/issues/3267
+	CHECKS["systemd"]=
+	# TODO https://github.com/lima-vm/lima/issues/3268
+	CHECKS["proxy-settings"]=
+	CHECKS["port-forwards"]=
+	;;
 esac
 
 if limactl ls -q | grep -q "$NAME"; then
@@ -80,7 +101,7 @@ fi
 # TODO: skip downloading and converting the image here.
 # Probably `limactl create` should have "dry run" mode that just generates `lima.yaml`.
 # shellcheck disable=SC2086
-"${LIMACTL_CREATE[@]}" ${LIMACTL_CREATE_ARGS} --set ".additionalDisks=null" --name="${NAME}-tmp" "$FILE"
+"${LIMACTL_CREATE[@]}" ${LIMACTL_CREATE_ARGS} --set ".additionalDisks=null" --name="${NAME}-tmp" "$FILE_HOST"
 case "$(yq '.networks[].lima' "${LIMA_HOME}/${NAME}-tmp/lima.yaml")" in
 "shared")
 	CHECKS["vmnet"]=1
@@ -93,32 +114,38 @@ esac
 limactl rm -f "${NAME}-tmp"
 
 if [[ -n ${CHECKS["port-forwards"]} ]]; then
-	tmpconfig="$HOME/lima-config-tmp"
+	tmpconfig="$HOME_HOST/lima-config-tmp"
 	mkdir -p "${tmpconfig}"
 	defer "rm -rf \"$tmpconfig\""
 	tmpfile="${tmpconfig}/${NAME}.yaml"
 	cp "$FILE" "${tmpfile}"
 	FILE="${tmpfile}"
+	FILE_HOST=$FILE
+	if [ "${OS_HOST}" = "Msys" ]; then
+		FILE_HOST="$(cygpath -w "$FILE")"
+	fi
+
 	INFO "Setup port forwarding rules for testing in \"${FILE}\""
 	"${scriptdir}/test-port-forwarding.pl" "${FILE}"
-	limactl validate "$FILE"
+	INFO "Validating \"$FILE_HOST\""
+	limactl validate "$FILE_HOST"
 fi
 
 function diagnose() {
 	NAME="$1"
 	set -x +e
-	tail "$HOME/.lima/${NAME}"/*.log
+	tail "$HOME_HOST/.lima/${NAME}"/*.log
 	limactl shell "$NAME" systemctl --no-pager status
 	limactl shell "$NAME" systemctl --no-pager
 	mkdir -p failure-logs
-	cp -pf "$HOME/.lima/${NAME}"/*.log failure-logs/
+	cp -pf "$HOME_HOST/.lima/${NAME}"/*.log failure-logs/
 	limactl shell "$NAME" sudo cat /var/log/cloud-init-output.log | tee failure-logs/cloud-init-output.log
 	set +x -e
 }
 
 export ftp_proxy=http://localhost:2121
 
-INFO "Creating \"$NAME\" from \"$FILE\""
+INFO "Creating \"$NAME\" from \"$FILE_HOST\""
 defer "limactl delete -f \"$NAME\""
 
 if [[ -n ${CHECKS["disk"]} ]]; then
@@ -130,7 +157,7 @@ fi
 
 set -x
 # shellcheck disable=SC2086
-"${LIMACTL_CREATE[@]}" ${LIMACTL_CREATE_ARGS} "$FILE"
+"${LIMACTL_CREATE[@]}" ${LIMACTL_CREATE_ARGS} "$FILE_HOST"
 set +x
 
 if [[ -n ${CHECKS["mount-path-with-spaces"]} ]]; then
@@ -152,7 +179,7 @@ limactl shell "$NAME" cat /etc/os-release
 set +x
 
 INFO "Testing that host home is not wiped out"
-[ -e "$HOME/.lima" ]
+[ -e "$HOME_HOST/.lima" ]
 
 if [[ -n ${CHECKS["mount-path-with-spaces"]} ]]; then
 	INFO 'Testing that "/tmp/lima test dir with spaces" is not wiped out'
@@ -179,16 +206,18 @@ if [[ -n ${CHECKS["set-user"]} ]]; then
 	limactl shell "$NAME" grep "^john:x:4711:4711:John Doe:/home/john-john" /etc/passwd
 fi
 
-INFO "Testing proxy settings are imported"
-got=$(limactl shell "$NAME" env | grep FTP_PROXY)
-# Expected: FTP_PROXY is set in addition to ftp_proxy, localhost is replaced
-# by the gateway address, and the value is set immediately without a restart
-gatewayIp=$(limactl shell "$NAME" ip route show 0.0.0.0/0 dev eth0 | cut -d\  -f3)
-expected="FTP_PROXY=http://${gatewayIp}:2121"
-INFO "FTP_PROXY: expected=${expected} got=${got}"
-if [ "$got" != "$expected" ]; then
-	ERROR "proxy environment variable not set to correct value"
-	exit 1
+if [[ -n ${CHECKS["proxy-settings"]} ]]; then
+	INFO "Testing proxy settings are imported"
+	got=$(limactl shell "$NAME" env | grep FTP_PROXY)
+	# Expected: FTP_PROXY is set in addition to ftp_proxy, localhost is replaced
+	# by the gateway address, and the value is set immediately without a restart
+	gatewayIp=$(limactl shell "$NAME" ip route show 0.0.0.0/0 dev eth0 | cut -d\  -f3)
+	expected="FTP_PROXY=http://${gatewayIp}:2121"
+	INFO "FTP_PROXY: expected=${expected} got=${got}"
+	if [ "$got" != "$expected" ]; then
+		ERROR "proxy environment variable not set to correct value"
+		exit 1
+	fi
 fi
 
 INFO "Testing limactl copy command"
@@ -196,6 +225,7 @@ tmpdir="$(mktemp -d "${TMPDIR:-/tmp}"/lima-test-templates.XXXXXX)"
 defer "rm -rf \"$tmpdir\""
 tmpfile="$tmpdir/lima-hostname"
 rm -f "$tmpfile"
+# TODO support Windows path https://github.com/lima-vm/lima/issues/3215
 limactl cp "$NAME":/etc/hostname "$tmpfile"
 expected="$(limactl shell "$NAME" cat /etc/hostname)"
 got="$(cat "$tmpfile")"
@@ -234,32 +264,38 @@ nginx_image="ghcr.io/stargz-containers/nginx:1.19-alpine-org"
 alpine_image="ghcr.io/containerd/alpine:3.14.0"
 
 if [[ -n ${CHECKS["container-engine"]} ]]; then
+	sudo=""
+	# Currently WSL2 machines only support privileged engine. This requirement might be lifted in the future.
+	if [[ "$(limactl ls --json "${NAME}" | jq -r .vmType)" == "wsl2" ]]; then
+		sudo="sudo"
+	fi
 	INFO "Run a nginx container with port forwarding 127.0.0.1:8080"
 	set -x
-	if ! limactl shell "$NAME" $CONTAINER_ENGINE info; then
-		limactl shell "$NAME" sudo cat /var/log/cloud-init-output.log
+	if ! limactl shell "$NAME" $sudo $CONTAINER_ENGINE info; then
+		limactl shell "$NAME" cat /var/log/cloud-init-output.log
 		ERROR "\"${CONTAINER_ENGINE} info\" failed"
 		exit 1
 	fi
-	limactl shell "$NAME" $CONTAINER_ENGINE pull --quiet ${nginx_image}
-	limactl shell "$NAME" $CONTAINER_ENGINE run -d --name nginx -p 127.0.0.1:8080:80 ${nginx_image}
+	limactl shell "$NAME" $sudo $CONTAINER_ENGINE pull --quiet ${nginx_image}
+	limactl shell "$NAME" $sudo $CONTAINER_ENGINE run -d --name nginx -p 127.0.0.1:8080:80 ${nginx_image}
 
 	timeout 3m bash -euxc "until curl -f --retry 30 --retry-connrefused http://127.0.0.1:8080; do sleep 3; done"
 
-	limactl shell "$NAME" $CONTAINER_ENGINE rm -f nginx
+	limactl shell "$NAME" $sudo $CONTAINER_ENGINE rm -f nginx
 	set +x
 	if [[ -n ${CHECKS["mount-home"]} ]]; then
-		hometmp="$HOME/lima-container-engine-test-tmp"
+		hometmp="$HOME_HOST/lima-container-engine-test-tmp"
+		hometmpguest="$HOME_GUEST/lima-container-engine-test-tmp"
 		# test for https://github.com/lima-vm/lima/issues/187
 		INFO "Testing home bind mount (\"$hometmp\")"
 		rm -rf "$hometmp"
 		mkdir -p "$hometmp"
 		defer "rm -rf \"$hometmp\""
 		set -x
-		limactl shell "$NAME" $CONTAINER_ENGINE pull --quiet ${alpine_image}
+		limactl shell "$NAME" $sudo $CONTAINER_ENGINE pull --quiet ${alpine_image}
 		echo "random-content-${RANDOM}" >"$hometmp/random"
 		expected="$(cat "$hometmp/random")"
-		got="$(limactl shell "$NAME" $CONTAINER_ENGINE run --rm -v "$hometmp/random":/mnt/foo ${alpine_image} cat /mnt/foo)"
+		got="$(limactl shell "$NAME" $sudo $CONTAINER_ENGINE run --rm -v "$hometmpguest/random":/mnt/foo ${alpine_image} cat /mnt/foo)"
 		INFO "$hometmp/random: expected=${expected}, got=${got}"
 		if [ "$got" != "$expected" ]; then
 			ERROR "Home directory is not shared?"
@@ -284,6 +320,9 @@ if [[ -n ${CHECKS["port-forwards"]} ]]; then
 	if [ "${NAME}" = "opensuse" ]; then
 		limactl shell "$NAME" sudo zypper in -y netcat-openbsd
 	fi
+	if limactl shell "$NAME" command -v dnf; then
+		limactl shell "$NAME" sudo dnf install -y nc
+	fi
 	"${scriptdir}/test-port-forwarding.pl" "${NAME}"
 
 	if [[ -n ${CHECKS["container-engine"]} || ${NAME} == "alpine"* ]]; then
@@ -306,6 +345,10 @@ if [[ -n ${CHECKS["port-forwards"]} ]]; then
 				rm nerdctl-full.tgz
 				sudo="sudo"
 			fi
+			# Currently WSL2 machines only support privileged engine. This requirement might be lifted in the future.
+			if [[ "$(limactl ls --json "${NAME}" | jq -r .vmType)" == "wsl2" ]]; then
+				sudo="sudo"
+			fi
 			limactl shell "$NAME" $sudo $CONTAINER_ENGINE info
 			limactl shell "$NAME" $sudo $CONTAINER_ENGINE pull --quiet ${nginx_image}
 
@@ -360,7 +403,8 @@ if [[ -n ${CHECKS["restart"]} ]]; then
 	fi
 
 	INFO "Stopping \"$NAME\""
-	limactl stop "$NAME"
+	# TODO https://github.com/lima-vm/lima/issues/3221
+	limactl stop "$NAME" || [ "${OS_HOST}" = "Msys" ]
 	sleep 3
 
 	if [[ -n ${CHECKS["disk"]} ]]; then
@@ -406,7 +450,7 @@ fi
 if [[ -n ${CHECKS["user-v2"]} ]]; then
 	INFO "Testing user-v2 network"
 	secondvm="$NAME-1"
-	"${LIMACTL_CREATE[@]}" --set ".additionalDisks=null" "$FILE" --name "$secondvm"
+	"${LIMACTL_CREATE[@]}" --set ".additionalDisks=null" "$FILE_HOST" --name "$secondvm"
 	if ! limactl start "$secondvm"; then
 		ERROR "Failed to start \"$secondvm\""
 		diagnose "$secondvm"
@@ -474,7 +518,8 @@ if [[ $NAME == "fedora" && "$(limactl ls --json "$NAME" | jq -r .vmType)" == "vz
 fi
 
 INFO "Stopping \"$NAME\""
-limactl stop "$NAME"
+# TODO https://github.com/lima-vm/lima/issues/3221
+limactl stop "$NAME" || [ "${OS_HOST}" = "Msys" ]
 sleep 3
 
 INFO "Deleting \"$NAME\""