Repository: lima-vm/lima
Pull request #726 (2022-03-13 09:14:15)

Add virtfs/9p mounts, instead of sshocker/sshfs
This PR allows selecting mount type, as "9p"

The default mount type is still as it was before.

~~Add mount option variable, for "rw" vs "ro"~~

Add mount type, for "reverse-sshfs" vs "9p"

Issue #20

~~QEMU with 9p-darwin patches available here:~~
~~https://github.com/afbjorklund/homebrew-core/blob/qemu-9p-darwin/Formula/qemu.rb~~
~~Patches from: https://github.com/willcohen/qemu/commits/v6.2.0-9p-darwin (or use HEAD)~~
~~You can also use the latest/greatest QEMU HEAD.~~

EDIT: The `qemu` in `brew` now supports virtfs, also for darwin systems.

----

Example mount when booting with "sshfs":

`:/tmp/lima on /tmp/lima type fuse.sshfs (rw,nosuid,nodev,relatime,user_id=1000,group_id=1000,allow_other)`

Example mount when booting with "9p":

`mount1 on /tmp/lima type 9p (rw,sync,dirsync,relatime,access=client,msize=131072,trans=virtio)`


afbjorklund: It is possible that it should not be called "mountType", and that the values should not be "sshfs" and "9p"

Those are the technical implementation details, it could be called "mountDriver" or whatever

And the human readable choices could be sshocker and virtfs, rather than the file system types.
afbjorklund: In order to test this on the Mac, we need to either patch qemu in brew or wait until the PR has been merged
afbjorklund: Instead of having to transport all the mount variables in environment variables,
maybe it would be better to run the mount over ssh - similar to the ReverseSSHFS

In the initial PoC, I just added the `mount` directly after the `mkdir -p` of the mountpoint

And the hostagent calls are basically no-op, instead of running ssh like with sshfs

There is no long-lived host process (but qemu) keeping the ssh connection open, though.

The ssh command would terminate, right after the `mount` (or the `umount` is complete.


chancez: I tried to test this on my m1 Mac using qemu with patches from https://github.com/Homebrew/homebrew-core/pull/96655/  but got an error:

```
(⎈ |rancher-desktop:default) ~/p/lima ❯❯❯ which qemu-system-aarch64                                                                                                                                                                                                                                                                           virtfs
/opt/homebrew/bin/qemu-system-aarch64
(⎈ |rancher-desktop:default) ~/p/lima ❯❯❯ ls -l `which qemu-system-aarch64`                                                                                                                                                                                                                                                                   virtfs
lrwxr-xr-x  1 chancezibolski  admin  46 Mar 13 11:30 /opt/homebrew/bin/qemu-system-aarch64 -> ../Cellar/qemu/6.2.0_1/bin/qemu-system-aarch64
(⎈ |rancher-desktop:default) ~/p/lima ❯❯❯ less ~/.lima/default/ha.stderr.log                                                                                                                                                                                                                                                                  virtfs
{"level":"debug","msg":"firmware candidates = [/opt/homebrew/share/qemu/edk2-aarch64-code.fd /usr/share/AAVMF/AAVMF_CODE.fd /usr/share/qemu-efi-aarch64/QEMU_EFI.fd]","time":"2022-03-13T14:58:20-07:00"}
{"level":"debug","msg":"OpenSSH version 8.6.1 detected","time":"2022-03-13T14:58:20-07:00"}
{"level":"debug","msg":"AES accelerator seems available, prioritizing aes128-gcm@openssh.com and aes256-gcm@openssh.com","time":"2022-03-13T14:58:20-07:00"}
{"level":"info","msg":"Starting QEMU (hint: to watch the boot progress, see \"/Users/chancezibolski/.lima/default/serial.log\")","time":"2022-03-13T14:58:20-07:00"}
{"level":"debug","msg":"qCmd.Args: [/opt/homebrew/bin/qemu-system-aarch64 -cpu host -machine virt,accel=hvf,highmem=off -smp 4,sockets=1,cores=4,threads=1 -m 4096 -drive if=pflash,format=raw,readonly=on,file=/opt/homebrew/share/qemu/edk2-aarch64-code.fd -boot order=c,splash-time=0,menu=on -drive file=/Users/chancezibolski/.lima/default/dif
{"level":"debug","msg":"qemu[stderr]: qemu-system-aarch64: -virtfs local,path=/Users/chancezibolski,mount_tag=mount0,security_model=mapped-xattr,readonly: warning: short-form boolean option 'readonly' deprecated","time":"2022-03-13T14:58:20-07:00"}
{"level":"debug","msg":"qemu[stderr]: Please use readonly=on instead","time":"2022-03-13T14:58:20-07:00"}
{"level":"info","msg":"Waiting for the essential requirement 1 of 3: \"ssh\"","time":"2022-03-13T14:58:20-07:00"}
{"level":"debug","msg":"qemu[stderr]: qemu-system-aarch64: -virtfs local,path=/tmp/lima,mount_tag=mount1,security_model=mapped-xattr: cannot initialize fsdev 'mount1': failed to open '/tmp/lima': No such file or directory","time":"2022-03-13T14:58:20-07:00"}
{"level":"debug","msg":"executing script \"ssh\"","time":"2022-03-13T14:58:20-07:00"}
{"error":"exit status 1","level":"info","msg":"QEMU has exited","time":"2022-03-13T14:58:20-07:00"}
{"level":"debug","msg":"executing ssh for script \"ssh\": /usr/bin/ssh [ssh -F /dev/null -o IdentityFile=\"/Users/chancezibolski/.lima/_config/user\" -o IdentityFile=\"/Users/chancezibolski/.ssh/id_rsa\" -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o NoHostAuthenticationForLocalhost=yes -o GSSAPIAuthentication=no -o Preferr
{"level":"debug","msg":"stdout=\"\", stderr=\"ssh: connect to host 127.0.0.1 port 60022: Connection refused\\r\\n\", err=failed to execute script \"ssh\": stdout=\"\", stderr=\"ssh: connect to host 127.0.0.1 port 60022: Connection refused\\r\\n\": exit status 255","time":"2022-03-13T14:58:20-07:00"}
```
afbjorklund: > Seems like perhaps lima was creating the mount location if it didn't exist previously?

Yeah, it was created too late.
afbjorklund: The 9p-darwin support has been merged in Homebrew now...

I will probably rename it to "virtfs", to match the "reverse-sshfs"
AkihiroSuda: > I will probably rename it to "virtfs", to match the "reverse-sshfs"

I'd prefer not, as "virtfs" is confusing with "virtiofs" https://virtio-fs.gitlab.io/
afbjorklund: Okay, but that's on them (they choose that confusing name in purpose, well knowing)

Will make another PR tonight, with the config options (but those _will_ be called "VirtFS")
AkihiroSuda: > Maybe virtfs-9p or so ?
"virtio-9p-pci" is the least confusing name, but maybe too much verbose.
https://github.com/qemu/qemu/blob/master/hw/virtio/virtio-9p-pci.c

"9p" would be fine for now.

willcohen: In terms of testing, after running creating `limactl start experimental-9p.yaml` and running `limactl shell experimental-9p`, where should the example directory be visible?
afbjorklund: The default exports your home directory as read-only, and `/tmp/lima` as read-write.

It's in the YAML, but you can also check with the `mount` (or `findmnt`) command
afbjorklund: Basically it only added one line. Too bad you can't "inherit" or include ?

`mountType: 9p`
jandubois: > Basically it only added one line. Too bad you can't "inherit" or include ?
> 
> `mountType: 9p`

I don't understand what you are saying here.
afbjorklund: The "experimental-9p.yaml" us the same as "default.yaml", except that mountType was changed from "reverse-sshfs" to "9p".

I plan to update the comments in it and add it to CI, next...
willcohen: It was just some silly user error on my end.

Mount output off of this branch confirms this now works using 9p:

```
$ mount
...
mount0 on /Users/wcohen type 9p (ro,relatime,sync,dirsync,access=client,msize=131072,trans=virtio)
mount1 on /tmp/lima type 9p (rw,relatime,sync,dirsync,access=client,msize=131072,trans=virtio)
```
matejsp: I have tried this patch on Mac (Montenery) with my patched colima. The only thing that I did was specify mount-type: 9p and restarted the qemu.

What is disappointing is that performance for r/w is a lot worse.

Running our python tests went from 15 seconds to 1 minute and 22 seconds. Docker desktop performance was 6 seconds.

```
arch: x86_64
images:
    - location: https://github.com/abiosoft/alpine-lima/releases/download/colima-v0.3.4-1/alpine-lima-clm-3.14.3-aarch64.iso
      arch: aarch64
      digest: sha512:363baa91e4087dfd04ec5eebadcb29b9aef45c2663642f951105b3989e93143ce45f94ba9101c01c5db46c3fc6b601340b39baad29b1a48bb4f735790048daaa
    - location: https://github.com/abiosoft/alpine-lima/releases/download/colima-v0.3.4-1/alpine-lima-clm-3.14.3-x86_64.iso
      arch: x86_64
      digest: sha512:cd7ad0ef76088ea3d9f428e70fcddcbbcc72999568aaee0de953052299a01df251454fa5db3a0fdcfa70896bd152bc5d92f9ad81d682f3ec44cfbd2149ae3856
cpus: 6
memory: 8GiB
disk: 40GiB
mounts:
    - location: "~"
      writable: true
    - location: /tmp/colima
      writable: true
ssh:
    localPort: 0
    loadDotSSHPubKeys: false
    forwardAgent: false
containerd:
    system: false
    user: false
firmware:
    legacyBIOS: false
hostResolver:
    enabled: true
    hosts:
        host.docker.internal: host.lima.internal
portForwards:
    - guestPortRange:
        - 0
        - 0
      guestSocket: /var/run/docker.sock
      hostPortRange:
        - 0
        - 0
      hostSocket: /Users/myuser/.colima/docker.sock
      proto: tcp
    - guestIPMustBeZero: true
      guestIP: 0.0.0.0
      guestPortRange:
        - 1
        - 65535
      hostIP: 0.0.0.0
      hostPortRange:
        - 1
        - 65535
      proto: tcp
    - guestIP: 127.0.0.1
      guestPortRange:
        - 1
        - 65535
      hostIP: 127.0.0.1
      hostPortRange:
        - 1
        - 65535
      proto: tcp
```

mount before
```
:/Users/myuser on /Users/myuser type fuse.sshfs (rw,nosuid,nodev,relatime,user_id=430655780,group_id=1000,allow_other)
:/tmp/colima on /tmp/colima type fuse.sshfs (rw,nosuid,nodev,relatime,user_id=430655780,group_id=1000,allow_other)
```
mount after
```
mount0 on /Users/myuser type 9p (rw,sync,dirsync,relatime,access=client,msize=131072,trans=virtio)
mount1 on /tmp/colima type 9p (rw,sync,dirsync,relatime,access=client,msize=131072,trans=virtio)
```


matejsp: I have increased msize to 512 KiB but this didn't help. I have also changed caching to mmap and it didn't help.
I will try with other caching types. 

Somewhere I stumbled across that using remote fs + docker virtual fs can be problematic.
But just to keep in mind that sshfs can still be faster in some cases :) 

No need to hold PR because of this.
afbjorklund: Thanks for testing, good to know...
matejsp: With fscache it is better (like 17 seconds). But still (a bit) slower than the sshfs.
willcohen: If ssh defaults to cache on, is there a downside to this PR changing `DEFAULT_VIRTFS_CACHE` to fscache to match previous behavior, especially if it's regressing that much less in the case described above?
LionsAd: @willcohen I would not turn on `cache=fscache` at this point as it destroys all throughput and made my VM lock-up in my performance tests. Setting `cache=mmap` by default was safe in my testing however.
willcohen: Makes sense. I suppose once the dust settles this just ultimately will want to go into docs somewhere or this github debate will live into eternity whenever a new user hits an unexpected performance issue and isn’t sure about implications of the settings tweaks, even if the docs largely just point right back to the QEMU 9p docs…
LionsAd: Agree - where would be a good place to document that?
AkihiroSuda: > Agree - where would be a good place to document that?

We should create docs/mount.md
willcohen: To @afbjorklund's point above, I suspect we'll need to see what comes up when podman users start utilizing this more fully as well to know the implications of the various choices -- I see no reason why the performance characteristics of the various mount options will differ that much for lima compared to podman.
afbjorklund: It is possible to use `sshocker` and reverse sshfs also with podman, they just made it more trouble to set up by hiding the ssh connection parameters in a "connection"
willcohen: Ah right, I didn't even think about a apples-to-apples of sshfs versus 9p. The whole connection parameters thing seems like such a complicated workaround (not to mention one that doesn't line up well with things like docker compose) that I was thinking more like the considerations of the various caching and msize options. 

I assume that once something along the lines of https://github.com/containers/podman/pull/13453 gets address for podman (or another option that gets things working with compose) for 4.1, then users will start using this stuff in a lot more earnest on that side.
chancez: Another thing to consider about performance is that VirtioFS is always a possibility (as Docker Desktop has shown), so at some point that'll probably be the successor to 9p/sshfs as well.
AkihiroSuda: I'd like to merge this soon, but this still has to be updated to:
- Create `/etc/fstab` entries (via cloud-init yaml, probably) so that the mounts are performed in the very early boot step
- Remove mentioning "virtfs" to avoid causing confusion with virtio-fs
afbjorklund: This probably means we need to move the mount, from the current location (using ssh)
matejsp: This latest rebase is no longer working for me. I am no longer getting 9p mounts.

```
arch: x86_64
images:
    - location: https://github.com/abiosoft/alpine-lima/releases/download/colima-v0.3.4-1/alpine-lima-clm-3.14.3-aarch64.iso
      arch: aarch64
      digest: sha512:363baa91e4087dfd04ec5eebadcb29b9aef45c2663642f951105b3989e93143ce45f94ba9101c01c5db46c3fc6b601340b39baad29b1a48bb4f735790048daaa
    - location: https://github.com/abiosoft/alpine-lima/releases/download/colima-v0.3.4-1/alpine-lima-clm-3.14.3-x86_64.iso
      arch: x86_64
      digest: sha512:cd7ad0ef76088ea3d9f428e70fcddcbbcc72999568aaee0de953052299a01df251454fa5db3a0fdcfa70896bd152bc5d92f9ad81d682f3ec44cfbd2149ae3856
cpus: 6
memory: 8GiB
disk: 40GiB
mounts:
    - location: "~"
      writable: true
    - location: /tmp/colima
      writable: true
mountType: 9p
ssh:
    localPort: 0
    loadDotSSHPubKeys: false
    forwardAgent: false
containerd:
    system: false
    user: false
firmware:
    legacyBIOS: false
hostResolver:
    enabled: true
    hosts:
        host.docker.internal: host.lima.internal
portForwards:
    - guestPortRange:
        - 0
        - 0
      guestSocket: /var/run/docker.sock
      hostPortRange:
        - 0
        - 0
      hostSocket: /Users/myuser/.colima/docker.sock
      proto: tcp
    - guestIPMustBeZero: true
      guestIP: 0.0.0.0
      guestPortRange:
        - 1
        - 65535
      hostIP: 0.0.0.0
      hostPortRange:
        - 1
        - 65535
      proto: tcp
    - guestIP: 127.0.0.1
      guestPortRange:
        - 1
        - 65535
      hostIP: 127.0.0.1
      hostPortRange:
        - 1
        - 65535
      proto: tcp
```
afbjorklund: Any errors ?
matejsp: What do you mean by first boot? 
First version of this patch worked on my already provisioned image with sshfs.
After that I was able to dynamically switch between 9p and sshfs. That's how I was testing 9p functionality.
And this image is the same. Just without 9p mounting points. But when switching to sshfs it works again.

Will also try with new clean image if it works.

```
colima:/var/log$ cat /etc/fstab 
/dev/cdrom	/media/cdrom	iso9660	noauto,ro 0 0
/dev/usbdisk	/media/usb	vfat	noauto,ro 0 0
```

```
colima:~$ cat /var/log/cloud-init-output.log 
+ export 'LIMA_CIDATA_MNT=/mnt/lima-cidata'
+ '[' --local '!=' --local ]
+ ln -s /var/log/lima-init.log /var/log/cloud-init-output.log
+ LIMA_CIDATA_DEV=/dev/disk/by-label/cidata
+ mkdir -p -m 700 /mnt/lima-cidata
+ mount -o 'ro,mode=0700,dmode=0700,overriderockperm,exec,uid=0' /dev/disk/by-label/cidata /mnt/lima-cidata
+ . /mnt/lima-cidata/lima.env
+ LIMA_CIDATA_NAME=colima
+ LIMA_CIDATA_USER=lima
+ LIMA_CIDATA_UID=430655780
+ LIMA_CIDATA_MOUNTS=2
+ LIMA_CIDATA_MOUNTS_0_MOUNTPOINT=/Users/matej.spiller-muys
+ LIMA_CIDATA_MOUNTS_1_MOUNTPOINT=/tmp/colima
+ LIMA_CIDATA_MOUNTTYPE=9p
+ LIMA_CIDATA_CONTAINERD_USER=
+ LIMA_CIDATA_CONTAINERD_SYSTEM=
+ LIMA_CIDATA_SLIRP_DNS=192.168.5.3
+ LIMA_CIDATA_SLIRP_GATEWAY=192.168.5.2
+ LIMA_CIDATA_SLIRP_IP_ADDRESS=192.168.5.15
+ LIMA_CIDATA_UDP_DNS_LOCAL_PORT=51647
+ LIMA_CIDATA_TCP_DNS_LOCAL_PORT=53866
+ awk '/^local-hostname:/ {print $2}' /mnt/lima-cidata/meta-data
+ LIMA_CIDATA_HOSTNAME=lima-colima
+ hostname colima
+ LIMA_CIDATA_HOMEDIR=/home/lima.linux
+ useradd --home-dir /home/lima.linux --create-home --uid 430655780 lima
+ echo 'lima ALL=(ALL) NOPASSWD:ALL'
+ LIMA_CIDATA_SSHDIR=/home/lima.linux/.ssh
+ mkdir -p -m 700 /home/lima.linux/.ssh
+ awk '/ssh-authorized-keys/ {flag=1; next} /^ *$/ {flag=0} flag {sub(/^ +- /, ""); gsub("\"", ""); print $0}' /mnt/lima-cidata/user-data
+ id -g lima
+ LIMA_CIDATA_GID=1000
+ chown -R 430655780:1000 /home/lima.linux/.ssh
+ chmod 600 /home/lima.linux/.ssh/authorized_keys
+ mkdir -p /var/lib/lima-init
+ IP_RENAME=/var/lib/lima-init/ip-rename
+ ip -o link
+ awk -f /usr/bin/lima-network.awk /var/lib/lima-init/ip-link /mnt/lima-cidata/network-config
+ chmod +x /var/lib/lima-init/ip-rename
+ ip link
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether 52:55:55:bd:98:28 brd ff:ff:ff:ff:ff:ff
+ /var/lib/lima-init/ip-rename
+ ip link
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether 52:55:55:bd:98:28 brd ff:ff:ff:ff:ff:ff
+ awk -f- /mnt/lima-cidata/network-config
+ awk -f- /mnt/lima-cidata/network-config
+ awk '/nameservers:/{flag=1; next} /^[^ ]/{flag=0} flag {gsub("^ +- +", ""); print}' /mnt/lima-cidata/user-data
+ tr '\n' ' '
+ DNS='192.168.5.3 '
+ '[' -n '192.168.5.3 ' ]
+ sed -i '/export dns/a dns="192.168.5.3 "' /usr/share/udhcpc/default.script
+ export 'LIMA_CIDATA_MNT=/mnt/lima-cidata'
+ '['  '!=' --local ]
+ exec /mnt/lima-cidata/boot.sh
LIMA| Executing /mnt/lima-cidata/boot/00-modprobe.sh
Loading kernel module "fuse"
Loading kernel module "tun"
Loading kernel module "tap"
Loading kernel module "bridge"
Loading kernel module "veth"
Loading kernel module "ip_tables"
Loading kernel module "ip6_tables"
Loading kernel module "iptable_nat"
Loading kernel module "ip6table_nat"
Loading kernel module "iptable_filter"
Loading kernel module "ip6table_filter"
Loading kernel module "nf_tables"
Loading kernel module "x_tables"
Loading kernel module "xt_MASQUERADE"
Loading kernel module "xt_addrtype"
Loading kernel module "xt_comment"
Loading kernel module "xt_conntrack"
Loading kernel module "xt_mark"
Loading kernel module "xt_multiport"
Loading kernel module "xt_nat"
Loading kernel module "xt_tcpudp"
Loading kernel module "overlay"
LIMA| Executing /mnt/lima-cidata/boot/01-alpine-ash-as-bash.sh
LIMA| Executing /mnt/lima-cidata/boot/05-persistent-data-volume.sh
+ test -f /etc/alpine-release
+ DATADIRS='/etc /home /tmp /usr/local /var/lib'
+ awk '$2 == "/" {print $3}' /proc/mounts
+ '[' tmpfs '==' tmpfs ]
+ mkdir -p /mnt/data
+ '[' -e /dev/disk/by-label/data-volume ]
+ mount -t ext4 /dev/disk/by-label/data-volume /mnt/data
+ '[' -d /mnt/data/etc ]
+ mkdir -p /etc
+ mount --bind /mnt/data/etc /etc
+ '[' -d /mnt/data/home ]
+ mkdir -p /home
+ mount --bind /mnt/data/home /home
+ '[' -d /mnt/data/tmp ]
+ mkdir -p /tmp
+ mount --bind /mnt/data/tmp /tmp
+ '[' -d /mnt/data/usr/local ]
+ mkdir -p /usr/local
+ mount --bind /mnt/data/usr/local /usr/local
+ '[' -d /mnt/data/var/lib ]
+ mkdir -p /var/lib
+ mount --bind /mnt/data/var/lib /var/lib
LIMA| Executing /mnt/lima-cidata/boot/06-etc-hosts.sh
+ sed -i /host.lima.internal/d /etc/hosts
+ echo -e '192.168.5.2\thost.lima.internal'
LIMA| Executing /mnt/lima-cidata/boot/07-etc-environment.sh
+ '[' -e /etc/environment ]
+ sed -i '/#LIMA-START/,/#LIMA-END/d' /etc/environment
+ cat /mnt/lima-cidata/etc_environment
+ command -v loginctl
+ rm -f /run/lima-guest-agent.sock
+ cp /mnt/lima-cidata/meta-data /run/lima-ssh-ready
LIMA| Executing /mnt/lima-cidata/boot/08-shell-prompt.sh
+ '[' colima '=' default ]
LIMA| Executing /mnt/lima-cidata/boot/09-host-dns-setup.sh
+ command -v iptables
+ '[' -n 51647 ]
+ '[' 51647 -ne 0 ]
+ iptables-save
+ grep 'udp.*192.168.5.2:51647'
+ iptables -t nat -A PREROUTING -d 192.168.5.3 -p udp --dport 53 -j DNAT --to-destination 192.168.5.2:51647
+ iptables -t nat -A OUTPUT -d 192.168.5.3 -p udp --dport 53 -j DNAT --to-destination 192.168.5.2:51647
+ '[' -n 53866 ]
+ '[' 53866 -ne 0 ]
+ iptables-save
+ grep 'tcp.*192.168.5.2:53866'
+ iptables -t nat -A PREROUTING -d 192.168.5.3 -p tcp --dport 53 -j DNAT --to-destination 192.168.5.2:53866
+ iptables -t nat -A OUTPUT -d 192.168.5.3 -p tcp --dport 53 -j DNAT --to-destination 192.168.5.2:53866
LIMA| Executing /mnt/lima-cidata/boot/10-alpine-prep.sh
+ test -f /etc/alpine-release
+ BRANCH=edge
+ awk '-F=' '$1=="VERSION_ID" {print $2}' /etc/os-release
+ VERSION_ID=3.14.3
+ BRANCH=v3.14
+ URL=https://dl-cdn.alpinelinux.org/alpine/v3.14/main
+ grep -q '^https://dl-cdn.alpinelinux.org/alpine/v3.14/main$' /etc/apk/repositories
+ URL=https://dl-cdn.alpinelinux.org/alpine/v3.14/community
+ grep -q '^https://dl-cdn.alpinelinux.org/alpine/v3.14/community$' /etc/apk/repositories
+ usermod -p '*' lima
+ sed -i 's/AllowTcpForwarding no/AllowTcpForwarding yes/g' /etc/ssh/sshd_config
+ rc-service --ifstarted sshd reload
+ rc-service cgroups start
 * WARNING: cgroups has already been started
+ rc-update add acpid
 * rc-update: acpid already installed in runlevel `default'; skipping
+ rc-service acpid start
 * Starting busybox acpid ... [ ok ]
LIMA| Executing /mnt/lima-cidata/boot/11-colorterm-environment.sh
+ grep -q COLORTERM /etc/ssh/sshd_config
+ sed -i 's/^AcceptEnv LANG LC_\*$/AcceptEnv COLORTERM LANG LC_*/' /etc/ssh/sshd_config
+ '[' -f /sbin/openrc-init ]
+ rc-service --ifstarted sshd reload
LIMA| Executing /mnt/lima-cidata/boot/20-rootless-base.sh
+ command -v systemctl
+ exit 0
LIMA| Executing /mnt/lima-cidata/boot/25-guestagent-base.sh
+ '[' 9p '!=' 9p ]
+ install -m 755 /mnt/lima-cidata/lima-guestagent /usr/local/bin/lima-guestagent
+ '[' -f /sbin/openrc-init ]
+ cat
+ chmod 755 /etc/init.d/lima-guestagent
+ rc-update add lima-guestagent default
 * rc-update: lima-guestagent already installed in runlevel `default'; skipping
+ rc-service lima-guestagent start
 * Caching service dependencies ... [ ok ]
 * Starting lima-guestagent ... [ ok ]
LIMA| Executing /mnt/lima-cidata/boot/30-install-packages.sh
+ INSTALL_IPTABLES=0
+ '['  '=' 1 ]
+ '['  '=' 1 ]
+ '[' 51647 -ne 0 ]
+ INSTALL_IPTABLES=1
+ command -v apt-get
+ command -v dnf
+ command -v yum
+ command -v pacman
+ command -v zypper
+ command -v apk
+ pkgs=
+ '[' 9p '=' reverse-sshfs ]
+ '[' 1 '=' 1 ]
+ command -v iptables
+ '[' -n  ]
+ SETUP_DNS=0
+ '[' -n 51647 ]
+ '[' 51647 -ne 0 ]
+ SETUP_DNS=1
+ '[' -n 53866 ]
+ '[' 53866 -ne 0 ]
+ SETUP_DNS=1
+ '[' 1 '=' 1 ]
+ /mnt/lima-cidata/boot/09-host-dns-setup.sh
+ command -v iptables
+ '[' -n 51647 ]
+ '[' 51647 -ne 0 ]
+ iptables-save
+ grep 'udp.*192.168.5.2:51647'
-A PREROUTING -d 192.168.5.3/32 -p udp -m udp --dport 53 -j DNAT --to-destination 192.168.5.2:51647
-A OUTPUT -d 192.168.5.3/32 -p udp -m udp --dport 53 -j DNAT --to-destination 192.168.5.2:51647
+ '[' -n 53866 ]
+ '[' 53866 -ne 0 ]
+ iptables-save
+ grep 'tcp.*192.168.5.2:53866'
-A PREROUTING -d 192.168.5.3/32 -p tcp -m tcp --dport 53 -j DNAT --to-destination 192.168.5.2:53866
-A OUTPUT -d 192.168.5.3/32 -p tcp -m tcp --dport 53 -j DNAT --to-destination 192.168.5.2:53866
+ '[' 9p '=' reverse-sshfs ]
LIMA| Executing /mnt/lima-cidata/boot/40-install-containerd.sh
+ '['  '!=' 1 ]
+ '['  '!=' 1 ]
+ exit 0
LIMA| Exiting with code 0
```
afbjorklund: > What do you mean by first boot?

`cloud-init` runs for the first setup.

Previously it was using the `ssh` to run the mount command, so that means it also works on an existing machine.
And that you can add and remove mounts during running (without reboot), even if there is no command for it yet.

https://cloudinit.readthedocs.io/en/latest/topics/boot.html#first-boot-determination

----

If it runs correctly, there should be some lines added to the `/etc/fstab` similar to:

```
mount0	/home/anders	9p	ro,version=9p2000.L,msize=131072,cache=none,trans=virtio,comment=cloudconfig	0	0
mount1	/tmp/lima	9p	rw,version=9p2000.L,msize=131072,cache=none,trans=virtio,comment=cloudconfig	0	0
```

matejsp: So it is no longer possible to switch dynamically :( ... how (if) is it possible to run cloud-init on existing image?
afbjorklund: Podman Machine has the same setup, basically it needs that you need to re-create the VM to change mounts.
afbjorklund: @matejsp : I guess the old code could be restored, so that the disks would be mounted twice
matejsp: My take is well since 9p is a new functionality I would possibility to be enabled on existing VMs (like original PR).
We have a bunch of existing users with a bunch of existing VMs. I would offer them a way to test new functionality without destroying/recreating their work/image. Especially since this solution is not all around best for everybody.

Mounting already mounted is really minimal (perf) penalty imho. I vote to restore this functionality.
afbjorklund: It's not really a performance issue, can just check if it is already mounted - before trying to mount it again.
afbjorklund: It gets a _lot_ worse, if you move in the other direction. Now the virtfs devices can't be found, and the boot hangs.

```
INFO[0000] [hostagent] Waiting for the essential requirement 1 of 5: "ssh" 
```

Because now the filesystems to be mounted cannot be found, and the whole system goes into emergency mode.

```
[  OK  ] Reached target Local File Systems (Pre).
         Mounting /home/anders...
         Mounting Mount unit for core20, revision 1328...
         Mounting Mount unit for core20, revision 1376...
         Mounting Mount unit for lxd, revision 22292...
         Mounting Mount unit for lxd, revision 22754...
         Mounting Mount unit for snapd, revision 15177...
         Mounting /tmp/lima...
         Starting File System Check on /dev/disk/by-label/UEFI...
[  OK  ] Started File System Check Daemon to report status.
[FAILED] Failed to mount /tmp/lima.
See 'systemctl status tmp-lima.mount' for details.
[DEPEND] Dependency failed for Local File Systems.
[FAILED] Failed to mount /home/anders.
See 'systemctl status home-anders.mount' for details.
```

```
[  OK  ] Reached target Network is Online.
[  OK  ] Reached target Remote File Systems (Pre).
[  OK  ] Reached target Remote File Systems.
[  OK  ] Finished Availability of block devices.
You are in emergency mode. After logging in, type "journalctl -xb" to view
system logs, "systemctl reboot" to reboot, "systemctl default" or "exit"
to boot into default mode.
Press Enter for maintenance
(or press Control-D to continue): 
```
afbjorklund: For newly created VMs, the cloud-init will generate the fstab from the `mounts:` and systemd will mount them at boot...

This was more for existing VMs or for added mounts, then they will be mounted at the same time that the sshfs was before.
So it might be a bit later than when they _could_ be mounted, but it also shouldn't change anything from how it was previous.

Also added the `nofail` option, so that the VM will still be accessible even if the 9p mounts couldn't be found for some reason.
afbjorklund: Can we test experimental-9p.yaml in cirrus ? The GHA test seemed to time out after an hour ?

> The "Tier 1" yamls (marked with ⭐) are regularly tested on the CI.
jandubois: > For newly created VMs, the cloud-init will generate the fstab from the mounts: and systemd will mount them at boot...

@afbjorklund I'm not sure why we need a separate code path for existing VMs. We create a new `instance-id` in the `meta-data` file for cloud-init, so each boot looks like a "first boot" to cloud-init: https://github.com/lima-vm/lima/blob/e021ce73c544d4abaf4e2511a8631d98640a3595/pkg/cidata/cidata.go#L109-L110

If the duplicated code is truly necessary, then at least some parts, like the options construction should be factored out into a single method that is used in both places.

I'm out of time for today; will look some more tomorrow.
jandubois: > For newly created VMs, the cloud-init will generate the fstab from the mounts: and systemd will mount them at boot...

I realize that this currently doesn't work on our default Alpine image and will need some additional support in `lima-init` (unless you use the `ci` edition of the image, but not sure if that alone will be enough).
jandubois: @afbjorklund I've commented out virtually all of `setupMount9P`, and everything still works, including updating a VM from reverse-sshfs to 9p. So why is this function needed?

If this is solely for the benefit of alpine-lima with the lima-init service, then I would rather fix `lima-init` than have the redundant code here. Moving it to `lima-init` also has the advantage that the mounts are already in place when provisioning scripts run.
jandubois: @willcohen I do have problems with mounting additional volumes:

```console
$ ls -l ~/go
lrwxr-xr-x  1 jan  staff  24  7 Feb  2020 /Users/jan/go -> /Volumes/ThunderBlade/go

$ lima ls -l ~/go
ls: /Users/jan/go: Network dropped connection on reset
ls: cannot read symbolic link '/Users/jan/go': Timer expired
lrwxr-xr-x 1 jan dialout 24 Feb  7  2020 /Users/jan/go

$ lima ls -l /Volumes/ThunderBlade/go
ls: /Volumes/ThunderBlade/go: Network dropped connection on reset
ls: /Volumes/ThunderBlade/go/bin: Network dropped connection on reset
ls: /Volumes/ThunderBlade/go/pkg: Network dropped connection on reset
ls: /Volumes/ThunderBlade/go/src: Network dropped connection on reset
total 0
drwxr-xr-x 32 jan dialout 1088 Aug 17  2021 bin
drwxr-xr-x  6 jan dialout  204 Dec 11  2020 pkg
drwxr-xr-x 14 jan dialout  476 Dec 11  2020 src
```

I use these defaults:

```console
$ cat ~/.lima/_config/default.yaml
mounts:
- location: "~"
  writable: true
- location: /Volumes/ThunderBlade
  writable: true
```

I assume this is a problem with the 9p support in qemu and not this PR, but don't really know what is going on.
jandubois: > I do have problems with mounting additional volumes:

Just for completeness, these work fine with reverse-sshfs:

```console
$ lima ls -l ~/go
lrwxr-xr-x 1 jan dialout 24 Feb  7  2020 /Users/jan/go -> /Volumes/ThunderBlade/go

$ lima ls -l ~/go/
total 12
drwxr-xr-x 1 jan dialout 1088 Aug 17  2021 bin
drwxr-xr-x 1 jan dialout  204 Dec 11  2020 pkg
drwxr-xr-x 1 jan dialout  476 Dec 11  2020 src

$ lima ls -l /Volumes/ThunderBlade/go
total 12
drwxr-xr-x 1 jan dialout 1088 Aug 17  2021 bin
drwxr-xr-x 1 jan dialout  204 Dec 11  2020 pkg
drwxr-xr-x 1 jan dialout  476 Dec 11  2020 src
```
jandubois: I just noticed that my message above looks like accessing the external volume works, but just generates additional noise. That is not correct; it frequently doesn't work at all:

```console
$ lima ls ~/go
ls: cannot access '/Users/jan/go': Timer expired

$ lima ls ~/go/
ls: cannot access '/Users/jan/go/': Timer expired

$ lima ls -l ~/go
ls: /Users/jan/go: Network dropped connection on reset
ls: cannot read symbolic link '/Users/jan/go': Timer expired
lrwxr-xr-x 1 jan dialout 24 Feb  7  2020 /Users/jan/go

$ lima ls -l ~/go/
ls: cannot access '/Users/jan/go/': Timer expired
```
afbjorklund: > @afbjorklund I've commented out virtually all of `setupMount9P`, and everything still works, including updating a VM from reverse-sshfs to 9p. So why is this function needed?
> 
> If this is solely for the benefit of alpine-lima with the lima-init service, then I would rather fix `lima-init` than have the redundant code here. Moving it to `lima-init` also has the advantage that the mounts are already in place when provisioning scripts run.

It was removed earlier, I was under the impression it was still needed when changing an existing VM...

But like you mention, maybe cloud-init is indeed run at every boot for everything not using alpine-lima ?
abiosoft: > Hope that it will make it to the "colima" fork, as well ?

Yeah it will. Any changes in alpine-lima will end up in the fork as upstream is being monitored.
afbjorklund: It was also just a fallback, if everything was mounted at boot then the code should never be called...
jandubois: > > why is this function needed?
> 
> It was also just a fallback, if everything was mounted at boot then the code should never be called...

I would prefer not to have code that should never be called. We should generate an error message if volumes are not mounted correctly, and then investigate what happened, and why. So I would prefer that the "dead" code be removed, but let's wait to hear what @AkihiroSuda thinks!

I'll try to update `lima-init` to support the `mounts` section in `user-data`.

I would be interested to hear if other people can reproduce the problems I had when mounting an additional volume (and using symlinks between the boot volume and the additional volume).
afbjorklund: It is not _dead_, it is just only used either older ISO
willcohen: @jandubois I'm unfamiliar enough with `lima` internals that it's hard for me to fully assess where the bug described above lies. 

My current inclination without knowing more is that I'm not sure why it'd be happening in QEMU (though that's certainly possible!).

Here's what I can say so far:
1. 9p-darwin on QEMU passes all the "synth" tests (https://wiki.qemu.org/Documentation/9p#Synth_Tests), which to the best of my knowledge is a pretty comprehensive check of behavior correctness.
2. I can't replicate this behavior on vanilla QEMU. To check, boot a normal working Linux image on QEMU and mount two directories: 
```
qemu-system-foo ... -otheroptionshere ...  \
-virtfs local,path=/Users/myuser/foo,mount_tag=foo,security_model=mapped-xattr \
-virtfs local,path=/Users/myuser/foo2,mount_tag=foo2,security_model=mapped-xattr
``` 
From there, you should be able to do all the various actions within `./mnt-foo` and `./mnt-foo2`, including `ls` and `ls -l`, after mounting manually with: 
```
mkdir -p mnt-foo
mkdir -p mnt-foo2
sudo mount -t 9p -o trans=virtio,version=9p2000.L foo ./mnt-foo
sudo mount -t 9p -o trans=virtio,version=9p2000.L foo2 ./mnt-foo2
```

The only errors I see are with Mac symbolic links (Dropbox, etc) that I don't expect to resolve correctly when mounted externally.
willcohen: Also quoting from the QEMU 9p page (https://wiki.qemu.org/Documentation/9p):

> These are some of the things that we might want to change on 9p protocol level in future. Right now this list just serves for roughly collecting some ideas for future protocol changes. Don't expect protocol changes in near future though, this will definitely take a long time.
> 
> Fixes:
> Increase [qid.path](http://ericvh.github.io/9p-rfc/rfc9p2000.html#anchor32) Size: The qid.path (which should not be confused with a filesystem path like "/foo/bar/") is an integer supposed to uniquely identify a file, which is currently a 64-bit number. A filesystem on host often has things like hard links which means different pathes on the filesystem might actually point to the same file and a numeric file ID in general is used to detect that by systems. Certain services like Samba are using this information, and incorrect handling (i.e. collisions) of unique file IDs can cause misbehaviours. The problem though is that 9p might share more than one filesystem anywhere under its 9p share's root path. So a truly unique file ID under Linux for instance is the combination of the mounted filesystem's device ID and the individual file's inode number, which is larger than 64-bit combined and hence would exceed 9p protocol's qid.path field. By default we only pass the file's inode number via qid.path, so we are assuming that only one filesystem is shared per 9p share. If multiple filesystems are detected, a warning is logged at runtime noting that file ID collisions are possible, and suggesting to enable the multidevs=remap option, which (if enabled) remaps file IDs from host to guest in a way that would prevent such collisions. In practice this remapping should happen with no noticable overhead, but obviously in a future protocol change this should be addressed by simply increasing the qid.path e.g. to 128 bits so that we won't need to remap file IDs in future anymore.

If the external volume we're talking about is, say, mounted on Mac via samba, that could very well be the issue!
afbjorklund: It seems like an external drive (https://www.owcdigital.com/products/thunderblade)
willcohen: I'm all APFS these days so I admit that I wasn't running through HFS repeatedly as a use case while this was going through review. One of the things that went through the most revision was dealing with the lack of `d_seekoff` on macOS (tested more robustly with APFS, but I don't see why HFS should differ too much): https://lists.gnu.org/archive/html/qemu-devel/2022-01/msg05963.html

Final version of what got merged upstream: https://gitlab.com/qemu-project/qemu/-/commit/6b3b279bd670c6a2fa23c9049820c814f0e2c846
jandubois: > It is not dead, it is just only used with older ISO (like the problem above)

@afbjorklund Yeah, but it is easy to replace the old ISO with a new ISO; just copy it over the `basedisk`. Since you already have to update qemu and lima to use 9p, it doesn't feel unreasonable to also require an ISO update if you want to get it working with an old instance.

E.g. Rancher Desktop ships with all prerequisites (lima, qemu, iso) and will replace the ISO during startup.
jandubois: It turns out I get these warning on **all** mounts, not just the external volume:

```console
$ lima ls -l /Users
ls: /Users/jan: Network dropped connection on reset
total 0
drwxr-xr-x 143 jan dialout 4576 Apr  4 20:11 jan
```

Maybe it is related to the macOS version (running on Catalina right now). Will try on Monterey, to see if that makes a difference.
jandubois: > Maybe it is related to the macOS version (running on Catalina right now). Will try on Monterey, to see if that makes a difference.

I tried this on a separate machine, a Monterey 12.3.1 laptop (with qemu 6.2.0_1 installed via `brew`), and the result is the same:

```console
$ gh pr checkout 726
$ make all install
$ limactl start
# edit mountType to "9"
$ lima ls -l /Users
ls: /Users/jan: Network dropped connection on reset
total 0
drwxr-xr-x 73 jan dialout 2336 Apr  4 20:31 jan
```
jandubois: I've now tried this on a Big Sur 11.6.4 machine with an M1 chip, and the result is still the same:

```console
jan@zilicon lima % lima uname -a
Linux lima-default 5.13.0-28-generic #31-Ubuntu SMP Thu Jan 13 17:44:32 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux
jan@zilicon lima % lima ls -l /Users
ls: /Users/jan: Network dropped connection on reset
total 0
drwxr-xr-x 34 jan dialout 1088 Apr  4 20:57 jan
jan@zilicon lima % sw_vers
ProductName:	macOS
ProductVersion:	11.6.4
BuildVersion:	20G417
```

I don't understand why nobody else would see this? Could somebody please confirm that `lima ls -l /Users` does **not** display the network error for them?
jandubois: > But like you mention, maybe cloud-init is indeed run at every boot for everything not using alpine-lima ?
> 
> Adding support for `mounts:` to lima-init sounds good.

@afbjorklund:

I've implemented this in https://github.com/lima-vm/alpine-lima/pull/53/files and released new ISOs at https://github.com/lima-vm/alpine-lima/releases/tag/v0.2.10.

I've confirmed that mounts get added to `/etc/fstab` on each reboot, as expected, and not just during first boot.
jandubois: > It is not _dead_, it is just only used with older ISO (like the problem above)

@afbjorklund As I mentioned above, this is only an issue for old instances based on the lima-init based Alpine image. Every other image automatically updates `/etc/fstab` via cloud-init.

I've updated the alpine-lima image with a version of lima-init that works the same way; and you can copy the new ISO over your existing `basedisk` to get this functionality.

As an alternative, you can also put the corresponding code into a provisioning script (you have to edit `lima.yaml` anyways to set the `mountType`). I've tested this (which is literally a copy of the code from lima-init, plus `mkmntdirs; mount -a` because the `localmount` service has already started):

```yaml
mountType: 9p
provision:
- mode: system
  script: |
    #!/bin/sh
    sed -i '/#LIMA-START/,/#LIMA-END/d' /etc/fstab
    echo "#LIMA-START" >> /etc/fstab
    awk -f- "${LIMA_CIDATA_MNT}"/user-data <<'EOF' >> /etc/fstab
    /^mounts:/ {
        flag = 1
        next
    }
    /^ *$/ {
        flag = 0
    }
    flag {
        sub(/^ *- \[/, "")
        sub(/"?\] *$/, "")
        gsub("\"?, \"?", "\t")
        print $0
    }
    EOF
    echo "#LIMA-END" >> /etc/fstab
    mkmntdirs
    mount -a
```

I've created an alpine image with the release version of lima. Then I switched to this PR branch and disabled the `setupMount9P` function, compiled and installed it. I stopped and restarted the instance.

As expected, the mounts no longer worked. I stopped again, added the snippet above to `lima.yaml` and started again. The mount points all worked.

Well, worked as well (or better) than on Ubuntu, but I also get spurious errors on Alpine, e.g.

```console
$ l shell alpine
ash: cd: line 1: can't cd to /Users/jan/suse/lima: Device timeout
lima-alpine:/Users/jan$ ls
```

Anyways, there are now 2 mechanisms to add 9p support to old Alpine instances, so I feel that we don't need the redundant code inside Lima itself; the mount logic is already convoluted enough without it.

I would like to know what @AkihiroSuda and @abiosoft are thinking about this.
abiosoft: > Anyways, there are now 2 mechanisms to add 9p support to old Alpine instances, so I feel that we don't need the redundant code inside Lima itself; the mount logic is already convoluted enough without it.
> I would like to know what @AkihiroSuda and @abiosoft are thinking about this.

I have already synced alpine-lima and generated new isos and it worked fine. From my tests, it even fixed a popular volume issue with postgres (and similar) containers https://github.com/abiosoft/colima/issues/102#issuecomment-1010910354.

As for Colima, users need to upgrade to use 9p anyways and that comes with the new iso. Thereby, support for older Alpine instances is not a requirement.
AkihiroSuda: Me too, I don't prefer to complicate the stuff for supporting old Alpine.
I can't read awk 😅 
AkihiroSuda: > I would prefer not to have code that should never be called. We should generate an error message if volumes are not mounted correctly, and then investigate what happened, and why. So I would prefer that the "dead" code be removed, but let's wait to hear what @AkihiroSuda thinks!

Generating an error SGTM
jandubois: > I can't read awk 😅

The awk code just extracts the `mounts` section from `user-data`:

```yaml
mounts:
- [mount0, /Users/jan, 9p, "rw,trans=virtio,version=9p2000.L,msize=131072,cache=mmap,nofail", "0", "0"]
- [mount1, /Volumes/ThunderBlade, 9p, "rw,trans=virtio,version=9p2000.L,msize=131072,cache=mmap,nofail", "0", "0"]
- [mount2, /tmp/lima, 9p, "rw,trans=virtio,version=9p2000.L,msize=131072,cache=mmap,nofail", "0", "0"]
```

And transforms them into the format needed for `/etc/fstab`. It would be more concise and readable if we had `perl` in the image (ok, maybe just more concise for most 😂):

```console
lima-alpine:/mnt/lima-cidata# perl -nE 'if (/^mounts:/../^$/){s/"?, "?/\t/g; say $1 if /\[(.+)"\]/}' user-data
mount0	/Users/jan	9p	rw,trans=virtio,version=9p2000.L,msize=131072,cache=mmap,nofail	0	0
mount1	/Volumes/ThunderBlade	9p	rw,trans=virtio,version=9p2000.L,msize=131072,cache=mmap,nofail	0	0
mount2	/tmp/lima	9p	rw,trans=virtio,version=9p2000.L,msize=131072,cache=mmap,nofail	0	0
```
jandubois: Posting this here again, as it may have gotten lost in the deluge of messages above:

> I don't understand why nobody else would see this? Could somebody please confirm that lima ls -l /Users does not display the network error for them?

I've tested the `default` template with `mountType: 9p` on:

* Intel iMac with Catalina
* Intel Macbook with Monterey
* M1 Mini with Big Sur

and on all machines I got an error like this:

```console
$ lima ls -l /Users
ls: /Users/jan: Network dropped connection on reset
total 0
drwxr-xr-x 73 jan dialout 2336 Apr  4 20:31 jan
```

Could somebody else please confirm if they see the same error, or if the command works without problem for them!
abiosoft: > Posting this here again, as it may have gotten lost in the deluge of messages above:
> 
> > I don't understand why nobody else would see this? Could somebody please confirm that lima ls -l /Users does not display the network error for them?
> 
> I've tested the `default` template with `mountType: 9p` on:
> 
> * Intel iMac with Catalina
> * Intel Macbook with Monterey
> * M1 Mini with Big Sur
> 
> and on all machines I got an error like this:
> 
> ```
> $ lima ls -l /Users
> ls: /Users/jan: Network dropped connection on reset
> total 0
> drwxr-xr-x 73 jan dialout 2336 Apr  4 20:31 jan
> ```
> 
> Could somebody else please confirm if they see the same error, or if the command works without problem for them!

@jandubois Which Linux distro, Alpine or Ubuntu?
jandubois: > > ```
> > $ lima ls -l /Users
> > ls: /Users/jan: Network dropped connection on reset
> > total 0
> > drwxr-xr-x 73 jan dialout 2336 Apr  4 20:31 jan
> > ```
> > 
> > 
> >     
> >       
> >     
> > 
> >       
> >     
> > 
> >     
> >   
> > Could somebody else please confirm if they see the same error, or if the command works without problem for them!
> 
> 
> @jandubois Which Linux distro, Alpine or Ubuntu?

This is with the `default` template (Ubuntu), just setting `mountType: 9p` and no other change.

I also get some problems on Alpine, related to cross-device symlinks, but the basic mounts are not throwing any errors.
abiosoft: @jandubois I get same error with default, though the `ls` output is valid and it exited with `0`.

I was unable to reproduce it on Alpine.
afbjorklund: > Anyways, there are now 2 mechanisms to add 9p support to old Alpine instances, so I feel that we don't need the redundant code inside Lima itself; the mount logic is already convoluted enough without it.

I will revert (revert-revert-revert ?) the compatibility mount then, and just skip mounting if using virtfs.

https://github.com/lima-vm/lima/commit/a406f306a961df0ae440fe389567533d4ac57633#diff-41b0aaae7171635e6c96b3659dfaf1821412066d9a76e427704adbeeb79a5328R28
jandubois: > I was unable to reproduce it on Alpine.

With Alpine I observed problems with symlink, e.g.

```console
$ ln -s / foo

$ ls -l foo
lrwxr-xr-x  1 jan  staff  1  5 Apr 22:52 foo -> /

$ limactl shell alpine ls -l /etc/mtab
lrwxrwxrwx    1 root     root            12 Apr  5 04:30 /etc/mtab -> /proc/mounts

$ limactl shell alpine ls -l foo
ls: foo: cannot read link: Device timeout
lrwxr-xr-x    1 jan      dialout          1 Apr  6 05:52 foo

$ limactl shell alpine ls -l foo/
ls: foo/: Device timeout
```

Symlinks don't seem to work at all with Alpine and 9p. On Ubuntu I would get the error message, but the command seems to work. On Alpine I cannot read a symlink at all:

`foo` still has the symlink attribute: `lrwxr-xr-x` but the `ls -l` command doesn't show the link target.

@abiosoft Can you reproduce this issue?
afbjorklund: I think symlinks might have some issues with 9p2000.L, maybe see if it is better with 9p2000.u ?
jandubois: > I think symlinks might have some issues with 9p2000.L, maybe see if it is better with 9p2000.u ?

Nope:

```console
$ limactl shell alpine cat /etc/fstab | grep jan
mount0	/Users/jan	9p	rw,trans=virtio,version=9p2000.u,msize=131072,cache=mmap,nofail	0	0

$ limactl shell alpine ls -l foo
ls: foo: Device timeout
```

abiosoft:  > Symlinks don't seem to work at all with Alpine and 9p. On Ubuntu I would get the error message, but the command seems to work. On Alpine I cannot read a symlink at all:
> 
> `foo` still has the symlink attribute: `lrwxr-xr-x` but the `ls -l` command doesn't show the link target.
> 
> @abiosoft Can you reproduce this issue?

@jandubois Yes, I was able to reproduce it. 

Symlinks created on the host only works on the host and symlinks created in the guest only works in the guest.

I did notice extra attributes when trying to access a guest-created symlink on the host.

```
~
$ ls -l@ foo
-rw-------@ 1 abiola  staff  1 Apr  6 07:11 foo
        user.virtfs.gid 4
        user.virtfs.mode        2
        user.virtfs.uid 4
```
jandubois: > Yes, I was able to reproduce it.

@abiosoft Thank you! I thought I was jinxed, as the discussion here sounded like 9p was working fine for everybody but me. But this looks like it is not production ready yet, and the weird additional attributes created on the host make me wary of using it in `writable: true` mode. :(

I still think we should merge this PR once the remaining issues about the mount logic are resolved, but I would consider this still rather experimental.
jandubois: I guess the next step will be to try to reproduce these issues with plain qemu, just so we can report them upstream if they are reproducible there and not specific to lima. I probably won't have time for further testing until next week though.
afbjorklund: > the weird additional attributes created on the host

This is due to using `mapped-xattr`, you can store them in a separate file with `mapped-file` if you prefer.

So, it's a feature.



Patch:
diff --git a/README.md b/README.md
index aefc50db..d2f4fa9e 100644
--- a/README.md
+++ b/README.md
@@ -266,7 +266,6 @@ The current default spec:
 - Performance optimization
 - More guest distros
 - Windows hosts
-- [VirtFS to replace the current reverse sshfs (work has to be done on QEMU repo)](https://github.com/NixOS/nixpkgs/pull/122420)
 - [vsock](https://github.com/apple/darwin-xnu/blob/xnu-7195.81.3/bsd/man/man4/vsock.4) to replace SSH (work has to be done on QEMU repo)
 
 ## FAQs & Troubleshooting
diff --git a/cmd/limactl/shell.go b/cmd/limactl/shell.go
index ce136481..54b57b1a 100644
--- a/cmd/limactl/shell.go
+++ b/cmd/limactl/shell.go
@@ -36,7 +36,6 @@ func newShellCommand() *cobra.Command {
 
 	shellCmd.Flags().SetInterspersed(false)
 
-	shellCmd.Flags().String("shell", "", "shell interpreter, e.g. /bin/bash")
 	shellCmd.Flags().String("workdir", "", "working directory")
 	return shellCmd
 }
@@ -84,19 +83,19 @@ func shellAction(cmd *cobra.Command, args []string) error {
 		return err
 	}
 	if workDir != "" {
-		changeDirCmd = fmt.Sprintf("cd %s || exit 1", shellescape.Quote(workDir))
+		changeDirCmd = fmt.Sprintf("cd %q || exit 1", workDir)
 		// FIXME: check whether y.Mounts contains the home, not just len > 0
 	} else if len(y.Mounts) > 0 {
 		hostCurrentDir, err := os.Getwd()
 		if err == nil {
-			changeDirCmd = fmt.Sprintf("cd %s", shellescape.Quote(hostCurrentDir))
+			changeDirCmd = fmt.Sprintf("cd %q", hostCurrentDir)
 		} else {
 			changeDirCmd = "false"
 			logrus.WithError(err).Warn("failed to get the current directory")
 		}
 		hostHomeDir, err := os.UserHomeDir()
 		if err == nil {
-			changeDirCmd = fmt.Sprintf("%s || cd %s", changeDirCmd, shellescape.Quote(hostHomeDir))
+			changeDirCmd = fmt.Sprintf("%s || cd %q", changeDirCmd, hostHomeDir)
 		} else {
 			logrus.WithError(err).Warn("failed to get the home directory")
 		}
@@ -109,16 +108,7 @@ func shellAction(cmd *cobra.Command, args []string) error {
 	}
 	logrus.Debugf("changeDirCmd=%q", changeDirCmd)
 
-	shell, err := cmd.Flags().GetString("shell")
-	if err != nil {
-		return err
-	}
-	if shell == "" {
-		shell = `"$SHELL"`
-	} else {
-		shell = shellescape.Quote(shell)
-	}
-	script := fmt.Sprintf("%s ; exec %s --login", changeDirCmd, shell)
+	script := fmt.Sprintf("%s ; exec $SHELL --login", changeDirCmd)
 	if len(args) > 1 {
 		script += fmt.Sprintf(
 			" -c %s",
diff --git a/docs/internal.md b/docs/internal.md
index 8287aabe..89554f35 100644
--- a/docs/internal.md
+++ b/docs/internal.md
@@ -109,6 +109,7 @@ The volume label is "cidata", as defined by [cloud-init NoCloud](https://cloudin
 - `LIMA_CIDATA_UID`: the numeric UID
 - `LIMA_CIDATA_MOUNTS`: the number of the Lima mounts
 - `LIMA_CIDATA_MOUNTS_%d_MOUNTPOINT`: the N-th mount point of Lima mounts (N=0, 1, ...)
+- `LIMA_CIDATA_MOUNTTYPE`: the type of the Lima mounts ("reverse-sshfs", "9p", ...)
 - `LIMA_CIDATA_CONTAINERD_USER`: set to "1" if rootless containerd to be set up
 - `LIMA_CIDATA_CONTAINERD_SYSTEM`: set to "1" if system-wide containerd to be set up
 - `LIMA_CIDATA_SLIRP_GATEWAY`: set to the IP address of the host on the SLIRP network. `192.168.5.2`.
diff --git a/examples/README.md b/examples/README.md
index a87b94c2..399034e4 100644
--- a/examples/README.md
+++ b/examples/README.md
@@ -34,6 +34,7 @@ Container orchestration:
 Others:
 - [`vmnet.yaml`](./vmnet.yaml): ⭐enable [`vmnet.framework`](../docs/network.md)
 - [`deprecated/centos-7.yaml`](./deprecated/centos-7.yaml): [deprecated] CentOS 7
+- [`experimental-9p.yaml`](experimental-9p.yaml): use 9p mount type
 
 ## Tier 1
 
diff --git a/examples/default.yaml b/examples/default.yaml
index d2d7de06..c130cda8 100644
--- a/examples/default.yaml
+++ b/examples/default.yaml
@@ -61,11 +61,28 @@ mounts:
     # system will look and feel like regular files directories in the Guest OS.
     # 🟢 Builtin default: false
     followSymlinks: null
+  9p:
+    # Supported security models are "passthrough", "mapped-xattr", "mapped-file" and "none".
+    # 🟢 Builtin default: "mapped-xattr"
+    securityModel: null
+    # Select 9P protocol version. Valid options are: "9p2000" (legacy), "9p2000.u", "9p2000.L".
+    # 🟢 Builtin default: "9p2000.L"
+    protocolVersion: null
+    # The number of bytes to use for 9p packet payload, where 4KiB is the absolute minimum.
+    # 🟢 Builtin default: "128KiB"
+    msize: null
+    # Specifies a caching policy. Valid options are: "none", "loose", "fscache" and "mmap".
+    # 🟢 Builtin default: "mmap"
+    cache: null
 - location: "/tmp/lima"
   # 🟢 Builtin default: false
   # 🔵 This file: true (only for "/tmp/lima")
   writable: true
 
+# Mount type for above mounts, such as "reverse-sshfs" (from sshocker) or "9p" (EXPERIMENTAL, from QEMU’s virtio-9p-pci, aka virtfs)
+# 🟢 Builtin default: "reverse-sshfs"
+mountType: null
+
 ssh:
   # A localhost port of the host. Forwarded to port 22 of the guest.
   # 🟢 Builtin default: 0 (automatically assigned to a free port)
diff --git a/examples/experimental-9p.yaml b/examples/experimental-9p.yaml
new file mode 100644
index 00000000..05c78df8
--- /dev/null
+++ b/examples/experimental-9p.yaml
@@ -0,0 +1,23 @@
+# This example requires Lima v0.10.0 or later.
+# On macOS hosts, QEMU needs to be v7.0.0 or later. Homebrew's QEMU v6.2.0_1 can be used too.
+# This example is planned to be merged to default.yaml in Lima v1.0 (ETA: 2022 Q2).
+images:
+# Try to use release-yyyyMMdd image if available. Note that release-yyyyMMdd will be removed after several months.
+- location: "https://cloud-images.ubuntu.com/releases/21.10/release-20220201/ubuntu-21.10-server-cloudimg-amd64.img"
+  arch: "x86_64"
+  digest: "sha256:73fe1785c60edeb506f191affff0440abcc2de02420bb70865d51d0ff9b28223"
+- location: "https://cloud-images.ubuntu.com/releases/21.10/release-20220201/ubuntu-21.10-server-cloudimg-arm64.img"
+  arch: "aarch64"
+  digest: "sha256:1b5b3fe616e1eea4176049d434a360344a7d471f799e151190f21b0a27f0b424"
+# Fallback to the latest release image.
+# Hint: run `limactl prune` to invalidate the cache
+- location: "https://cloud-images.ubuntu.com/releases/21.10/release/ubuntu-21.10-server-cloudimg-amd64.img"
+  arch: "x86_64"
+- location: "https://cloud-images.ubuntu.com/releases/21.10/release/ubuntu-21.10-server-cloudimg-arm64.img"
+  arch: "aarch64"
+
+mounts:
+- location: "~"
+- location: "/tmp/lima"
+  writable: true
+mountType: "9p"
diff --git a/go.mod b/go.mod
index 4488d441..5428ffcf 100644
--- a/go.mod
+++ b/go.mod
@@ -7,7 +7,7 @@ require (
 	github.com/alessio/shellescape v1.4.1
 	github.com/cheggaaa/pb/v3 v3.0.8
 	github.com/containerd/containerd v1.6.2
-	github.com/containerd/continuity v0.3.0
+	github.com/containerd/continuity v0.2.2
 	github.com/coreos/go-semver v0.3.0
 	github.com/digitalocean/go-qemu v0.0.0-20210326154740-ac9e0b687001
 	github.com/diskfs/go-diskfs v1.2.0
@@ -28,13 +28,13 @@ require (
 	github.com/spf13/cobra v1.4.0
 	github.com/xorcare/pointer v1.1.0
 	github.com/yalue/native_endian v1.0.2
-	golang.org/x/sys v0.0.0-20220405210540-1e041c57c461
+	golang.org/x/sys v0.0.0-20220317061510-51cd9980dadf
 	gopkg.in/yaml.v2 v2.4.0
 	gotest.tools/v3 v3.1.0
 )
 
 require (
-	github.com/Microsoft/go-winio v0.5.2 // indirect
+	github.com/Microsoft/go-winio v0.5.1 // indirect
 	github.com/VividCortex/ewma v1.1.1 // indirect
 	github.com/digitalocean/go-libvirt v0.0.0-20201209184759-e2a69bcd5bd1 // indirect
 	github.com/fatih/color v1.13.0 // indirect
diff --git a/go.sum b/go.sum
index a5354ccb..f7325163 100644
--- a/go.sum
+++ b/go.sum
@@ -78,9 +78,8 @@ github.com/Microsoft/go-winio v0.4.17-0.20210211115548-6eac466e5fa3/go.mod h1:JP
 github.com/Microsoft/go-winio v0.4.17-0.20210324224401-5516f17a5958/go.mod h1:JPGBdM1cNvN/6ISo+n8V5iA4v8pBzdOpzfwIujj1a84=
 github.com/Microsoft/go-winio v0.4.17/go.mod h1:JPGBdM1cNvN/6ISo+n8V5iA4v8pBzdOpzfwIujj1a84=
 github.com/Microsoft/go-winio v0.5.0/go.mod h1:JPGBdM1cNvN/6ISo+n8V5iA4v8pBzdOpzfwIujj1a84=
+github.com/Microsoft/go-winio v0.5.1 h1:aPJp2QD7OOrhO5tQXqQoGSJc+DjDtWTGLOmNyAm6FgY=
 github.com/Microsoft/go-winio v0.5.1/go.mod h1:JPGBdM1cNvN/6ISo+n8V5iA4v8pBzdOpzfwIujj1a84=
-github.com/Microsoft/go-winio v0.5.2 h1:a9IhgEQBCUEk6QCdml9CiJGhAws+YwffDHEMp1VMrpA=
-github.com/Microsoft/go-winio v0.5.2/go.mod h1:WpS1mjBmmwHBEWmogvA2mj8546UReBk4v8QkMxJ6pZY=
 github.com/Microsoft/hcsshim v0.8.6/go.mod h1:Op3hHsoHPAvb6lceZHDtd9OkTew38wNoXnJs8iY7rUg=
 github.com/Microsoft/hcsshim v0.8.7-0.20190325164909-8abdbb8205e4/go.mod h1:Op3hHsoHPAvb6lceZHDtd9OkTew38wNoXnJs8iY7rUg=
 github.com/Microsoft/hcsshim v0.8.7/go.mod h1:OHd7sQqRFrYd3RmSgbgji+ctCwkbq2wbEYNSzOYtcBQ=
@@ -227,9 +226,8 @@ github.com/containerd/continuity v0.0.0-20200710164510-efbc4488d8fe/go.mod h1:cE
 github.com/containerd/continuity v0.0.0-20201208142359-180525291bb7/go.mod h1:kR3BEg7bDFaEddKm54WSmrol1fKWDU1nKYkgrcgZT7Y=
 github.com/containerd/continuity v0.0.0-20210208174643-50096c924a4e/go.mod h1:EXlVlkqNba9rJe3j7w3Xa924itAMLgZH4UD/Q4PExuQ=
 github.com/containerd/continuity v0.1.0/go.mod h1:ICJu0PwR54nI0yPEnJ6jcS+J7CZAUXrLh8lPo2knzsM=
+github.com/containerd/continuity v0.2.2 h1:QSqfxcn8c+12slxwu00AtzXrsami0MJb/MQs9lOLHLA=
 github.com/containerd/continuity v0.2.2/go.mod h1:pWygW9u7LtS1o4N/Tn0FoCFDIXZ7rxcMX7HX1Dmibvk=
-github.com/containerd/continuity v0.3.0 h1:nisirsYROK15TAMVukJOUyGJjz4BNQJBVsNvAXZJ/eg=
-github.com/containerd/continuity v0.3.0/go.mod h1:wJEAIwKOm/pBZuBd0JmeTvnLquTB1Ag8espWhkykbPM=
 github.com/containerd/fifo v0.0.0-20180307165137-3d5202aec260/go.mod h1:ODA38xgv3Kuk8dQz2ZQXpnv/UZZUHUCL7pnLehbXgQI=
 github.com/containerd/fifo v0.0.0-20190226154929-a9fb20d87448/go.mod h1:ODA38xgv3Kuk8dQz2ZQXpnv/UZZUHUCL7pnLehbXgQI=
 github.com/containerd/fifo v0.0.0-20191213151349-ff969a566b00/go.mod h1:jPQ2IAeZRCYxpS/Cm1495vGFww6ecHmMk1YJH2Q5ln0=
@@ -1273,9 +1271,8 @@ golang.org/x/sys v0.0.0-20210927094055-39ccf1dd6fa6/go.mod h1:oPkhp1MJrh7nUepCBc
 golang.org/x/sys v0.0.0-20211025201205-69cdffdb9359/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.0.0-20211116061358-0a5406a5449c/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.0.0-20211216021012-1d35b9e2eb4e/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/sys v0.0.0-20220317061510-51cd9980dadf h1:Fm4IcnUL803i92qDlmB0obyHmosDrxZWxJL3gIeNqOw=
 golang.org/x/sys v0.0.0-20220317061510-51cd9980dadf/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
-golang.org/x/sys v0.0.0-20220405210540-1e041c57c461 h1:kHVeDEnfKn3T238CvrUcz6KeEsFHVaKh4kMTt6Wsysg=
-golang.org/x/sys v0.0.0-20220405210540-1e041c57c461/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/term v0.0.0-20201117132131-f5c789dd3221/go.mod h1:Nr5EML6q2oocZ2LXRh80K7BxOlk5/8JxuGnuhpl+muw=
 golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
 golang.org/x/term v0.0.0-20210220032956-6a3ed077a48d/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
diff --git a/pkg/cidata/cidata.TEMPLATE.d/boot/25-guestagent-base.sh b/pkg/cidata/cidata.TEMPLATE.d/boot/25-guestagent-base.sh
index aa8c81b6..bada60bd 100644
--- a/pkg/cidata/cidata.TEMPLATE.d/boot/25-guestagent-base.sh
+++ b/pkg/cidata/cidata.TEMPLATE.d/boot/25-guestagent-base.sh
@@ -2,15 +2,17 @@
 
 set -eux
 
-# Create mount points
-# NOTE: Busybox sh does not support `for ((i=0;i<$N;i++))` form
-for f in $(seq 0 $((LIMA_CIDATA_MOUNTS - 1))); do
-	mountpointvar="LIMA_CIDATA_MOUNTS_${f}_MOUNTPOINT"
-	mountpoint="$(eval echo \$"$mountpointvar")"
-	mkdir -p "${mountpoint}"
-	gid=$(id -g "${LIMA_CIDATA_USER}")
-	chown "${LIMA_CIDATA_UID}:${gid}" "${mountpoint}"
-done
+if [ "${LIMA_CIDATA_MOUNTTYPE}" != "9p" ]; then
+	# Create mount points
+	# NOTE: Busybox sh does not support `for ((i=0;i<$N;i++))` form
+	for f in $(seq 0 $((LIMA_CIDATA_MOUNTS - 1))); do
+		mountpointvar="LIMA_CIDATA_MOUNTS_${f}_MOUNTPOINT"
+		mountpoint="$(eval echo \$"$mountpointvar")"
+		mkdir -p "${mountpoint}"
+		gid=$(id -g "${LIMA_CIDATA_USER}")
+		chown "${LIMA_CIDATA_UID}:${gid}" "${mountpoint}"
+	done
+fi
 
 # Install or update the guestagent binary
 install -m 755 "${LIMA_CIDATA_MNT}"/lima-guestagent /usr/local/bin/lima-guestagent
diff --git a/pkg/cidata/cidata.TEMPLATE.d/boot/30-install-packages.sh b/pkg/cidata/cidata.TEMPLATE.d/boot/30-install-packages.sh
index 8188745f..0f54defc 100644
--- a/pkg/cidata/cidata.TEMPLATE.d/boot/30-install-packages.sh
+++ b/pkg/cidata/cidata.TEMPLATE.d/boot/30-install-packages.sh
@@ -21,8 +21,10 @@ fi
 # Install minimum dependencies
 if command -v apt-get >/dev/null 2>&1; then
 	pkgs=""
-	if [ "${LIMA_CIDATA_MOUNTS}" -gt 0 ] && ! command -v sshfs >/dev/null 2>&1; then
-		pkgs="${pkgs} sshfs"
+	if [ "${LIMA_CIDATA_MOUNTTYPE}" = "reverse-sshfs" ]; then
+		if [ "${LIMA_CIDATA_MOUNTS}" -gt 0 ] && ! command -v sshfs >/dev/null 2>&1; then
+			pkgs="${pkgs} sshfs"
+		fi
 	fi
 	if [ "${INSTALL_IPTABLES}" = 1 ] && [ ! -e /usr/sbin/iptables ]; then
 		pkgs="${pkgs} iptables"
@@ -42,8 +44,10 @@ elif command -v dnf >/dev/null 2>&1; then
 	if ! command -v tar >/dev/null 2>&1; then
 		pkgs="${pkgs} tar"
 	fi
-	if [ "${LIMA_CIDATA_MOUNTS}" -gt 0 ] && ! command -v sshfs >/dev/null 2>&1; then
-		pkgs="${pkgs} fuse-sshfs"
+	if [ "${LIMA_CIDATA_MOUNTTYPE}" = "reverse-sshfs" ]; then
+		if [ "${LIMA_CIDATA_MOUNTS}" -gt 0 ] && ! command -v sshfs >/dev/null 2>&1; then
+			pkgs="${pkgs} fuse-sshfs"
+		fi
 	fi
 	if [ "${INSTALL_IPTABLES}" = 1 ] && [ ! -e /usr/sbin/iptables ]; then
 		pkgs="${pkgs} iptables"
@@ -102,8 +106,10 @@ elif command -v yum >/dev/null 2>&1; then
 	fi
 elif command -v pacman >/dev/null 2>&1; then
 	pkgs=""
-	if [ "${LIMA_CIDATA_MOUNTS}" -gt 0 ] && ! command -v sshfs >/dev/null 2>&1; then
-		pkgs="${pkgs} sshfs"
+	if [ "${LIMA_CIDATA_MOUNTTYPE}" = "reverse-sshfs" ]; then
+		if [ "${LIMA_CIDATA_MOUNTS}" -gt 0 ] && ! command -v sshfs >/dev/null 2>&1; then
+			pkgs="${pkgs} sshfs"
+		fi
 	fi
 	# other dependencies are preinstalled on Arch Linux
 	if [ -n "${pkgs}" ]; then
@@ -112,8 +118,10 @@ elif command -v pacman >/dev/null 2>&1; then
 	fi
 elif command -v zypper >/dev/null 2>&1; then
 	pkgs=""
-	if [ "${LIMA_CIDATA_MOUNTS}" -gt 0 ] && ! command -v sshfs >/dev/null 2>&1; then
-		pkgs="${pkgs} sshfs"
+	if [ "${LIMA_CIDATA_MOUNTTYPE}" = "reverse-sshfs" ]; then
+		if [ "${LIMA_CIDATA_MOUNTS}" -gt 0 ] && ! command -v sshfs >/dev/null 2>&1; then
+			pkgs="${pkgs} sshfs"
+		fi
 	fi
 	if [ "${INSTALL_IPTABLES}" = 1 ] && [ ! -e /usr/sbin/iptables ]; then
 		pkgs="${pkgs} iptables"
@@ -127,8 +135,10 @@ elif command -v zypper >/dev/null 2>&1; then
 	fi
 elif command -v apk >/dev/null 2>&1; then
 	pkgs=""
-	if [ "${LIMA_CIDATA_MOUNTS}" -gt 0 ] && ! command -v sshfs >/dev/null 2>&1; then
-		pkgs="${pkgs} sshfs"
+	if [ "${LIMA_CIDATA_MOUNTTYPE}" = "reverse-sshfs" ]; then
+		if [ "${LIMA_CIDATA_MOUNTS}" -gt 0 ] && ! command -v sshfs >/dev/null 2>&1; then
+			pkgs="${pkgs} sshfs"
+		fi
 	fi
 	if [ "${INSTALL_IPTABLES}" = 1 ] && ! command -v iptables >/dev/null 2>&1; then
 		pkgs="${pkgs} iptables"
@@ -154,4 +164,6 @@ fi
 
 # update_fuse_conf has to be called after installing all the packages,
 # otherwise apt-get fails with conflict
-update_fuse_conf
+if [ "${LIMA_CIDATA_MOUNTTYPE}" = "reverse-sshfs" ]; then
+	update_fuse_conf
+fi
diff --git a/pkg/cidata/cidata.TEMPLATE.d/lima.env b/pkg/cidata/cidata.TEMPLATE.d/lima.env
index 7c81c570..de3cfdf4 100644
--- a/pkg/cidata/cidata.TEMPLATE.d/lima.env
+++ b/pkg/cidata/cidata.TEMPLATE.d/lima.env
@@ -3,8 +3,9 @@ LIMA_CIDATA_USER={{ .User }}
 LIMA_CIDATA_UID={{ .UID }}
 LIMA_CIDATA_MOUNTS={{ len .Mounts }}
 {{- range $i, $val := .Mounts}}
-LIMA_CIDATA_MOUNTS_{{$i}}_MOUNTPOINT={{$val}}
+LIMA_CIDATA_MOUNTS_{{$i}}_MOUNTPOINT={{$val.Target}}
 {{- end}}
+LIMA_CIDATA_MOUNTTYPE={{ .MountType }}
 {{- if .Containerd.User}}
 LIMA_CIDATA_CONTAINERD_USER=1
 {{- else}}
diff --git a/pkg/cidata/cidata.TEMPLATE.d/user-data b/pkg/cidata/cidata.TEMPLATE.d/user-data
index 8570d846..ec34349f 100644
--- a/pkg/cidata/cidata.TEMPLATE.d/user-data
+++ b/pkg/cidata/cidata.TEMPLATE.d/user-data
@@ -5,6 +5,15 @@ growpart:
   mode: auto
   devices: ['/']
 
+{{- if eq .MountType "9p" }}
+{{- if .Mounts }}
+mounts:
+  {{- range $m := $.Mounts}}
+- [{{$m.Tag}}, {{$m.Target}}, {{$m.Type}}, "{{$m.Options}}", "0", "0"]
+  {{- end }}
+{{- end }}
+{{- end }}
+
 users:
   - name: "{{.User}}"
     uid: "{{.UID}}"
diff --git a/pkg/cidata/cidata.go b/pkg/cidata/cidata.go
index 98dff161..947ad957 100644
--- a/pkg/cidata/cidata.go
+++ b/pkg/cidata/cidata.go
@@ -11,6 +11,7 @@ import (
 	"strings"
 	"time"
 
+	"github.com/docker/go-units"
 	"github.com/lima-vm/lima/pkg/iso9660util"
 	"github.com/lima-vm/lima/pkg/limayaml"
 	"github.com/lima-vm/lima/pkg/localpathutil"
@@ -120,12 +121,44 @@ func GenerateISO9660(instDir, name string, y *limayaml.LimaYAML, udpDNSLocalPort
 		args.SSHPubKeys = append(args.SSHPubKeys, f.Content)
 	}
 
-	for _, f := range y.Mounts {
+	var fstype string
+	switch *y.MountType {
+	case limayaml.REVSSHFS:
+		fstype = "sshfs"
+	case limayaml.NINEP:
+		fstype = "9p"
+	}
+	for i, f := range y.Mounts {
+		tag := fmt.Sprintf("mount%d", i)
 		expanded, err := localpathutil.Expand(f.Location)
 		if err != nil {
 			return err
 		}
-		args.Mounts = append(args.Mounts, expanded)
+		options := "defaults"
+		if fstype == "9p" {
+			options = "ro"
+			if *f.Writable {
+				options = "rw"
+			}
+			options += ",trans=virtio"
+			options += fmt.Sprintf(",version=%s", *f.NineP.ProtocolVersion)
+			msize, err := units.RAMInBytes(*f.NineP.Msize)
+			if err != nil {
+				return fmt.Errorf("failed to parse msize for %q: %w", expanded, err)
+			}
+			options += fmt.Sprintf(",msize=%d", msize)
+			options += fmt.Sprintf(",cache=%s", *f.NineP.Cache)
+			// don't fail the boot, if virtfs is not available
+			options += ",nofail"
+		}
+		args.Mounts = append(args.Mounts, Mount{Tag: tag, Target: expanded, Type: fstype, Options: options})
+	}
+
+	switch *y.MountType {
+	case limayaml.REVSSHFS:
+		args.MountType = "reverse-sshfs"
+	case limayaml.NINEP:
+		args.MountType = "9p"
 	}
 
 	slirpMACAddress := limayaml.MACAddress(instDir)
diff --git a/pkg/cidata/template.go b/pkg/cidata/template.go
index 4a2cdb3f..a2e77029 100644
--- a/pkg/cidata/template.go
+++ b/pkg/cidata/template.go
@@ -27,13 +27,20 @@ type Network struct {
 	MACAddress string
 	Interface  string
 }
+type Mount struct {
+	Tag     string
+	Target  string // abs path, accessible by the User
+	Type    string
+	Options string
+}
 type TemplateArgs struct {
 	Name            string // instance name
 	IID             string // instance id
 	User            string // user name
 	UID             int
 	SSHPubKeys      []string
-	Mounts          []string // abs path, accessible by the User
+	Mounts          []Mount
+	MountType       string
 	Containerd      Containerd
 	Networks        []Network
 	SlirpNICName    string
@@ -62,7 +69,8 @@ func ValidateTemplateArgs(args TemplateArgs) error {
 	if len(args.SSHPubKeys) == 0 {
 		return errors.New("field SSHPubKeys must be set")
 	}
-	for i, f := range args.Mounts {
+	for i, m := range args.Mounts {
+		f := m.Target
 		if !filepath.IsAbs(f) {
 			return fmt.Errorf("field mounts[%d] must be absolute, got %q", i, f)
 		}
diff --git a/pkg/hostagent/hostagent.go b/pkg/hostagent/hostagent.go
index b5f4d8f3..bbf2e794 100644
--- a/pkg/hostagent/hostagent.go
+++ b/pkg/hostagent/hostagent.go
@@ -378,19 +378,21 @@ func (a *HostAgent) startHostAgentRoutines(ctx context.Context) error {
 	if err := a.waitForRequirements(ctx, "essential", a.essentialRequirements()); err != nil {
 		mErr = multierror.Append(mErr, err)
 	}
-	mounts, err := a.setupMounts(ctx)
-	if err != nil {
-		mErr = multierror.Append(mErr, err)
-	}
-	a.onClose = append(a.onClose, func() error {
-		var unmountMErr error
-		for _, m := range mounts {
-			if unmountErr := m.close(); unmountErr != nil {
-				unmountMErr = multierror.Append(unmountMErr, unmountErr)
-			}
+	if *a.y.MountType == limayaml.REVSSHFS {
+		mounts, err := a.setupMounts(ctx)
+		if err != nil {
+			mErr = multierror.Append(mErr, err)
 		}
-		return unmountMErr
-	})
+		a.onClose = append(a.onClose, func() error {
+			var unmountMErr error
+			for _, m := range mounts {
+				if unmountErr := m.close(); unmountErr != nil {
+					unmountMErr = multierror.Append(unmountMErr, unmountErr)
+				}
+			}
+			return unmountMErr
+		})
+	}
 	go a.watchGuestAgentEvents(ctx)
 	if err := a.waitForRequirements(ctx, "optional", a.optionalRequirements()); err != nil {
 		mErr = multierror.Append(mErr, err)
diff --git a/pkg/hostagent/requirements.go b/pkg/hostagent/requirements.go
index 8a89b9cc..8cf41d11 100644
--- a/pkg/hostagent/requirements.go
+++ b/pkg/hostagent/requirements.go
@@ -87,7 +87,7 @@ it must not be created until the session reset is done.
 `,
 		})
 
-	if len(a.y.Mounts) > 0 {
+	if *a.y.MountType == limayaml.REVSSHFS && len(a.y.Mounts) > 0 {
 		req = append(req, requirement{
 			description: "sshfs binary to be installed",
 			script: `#!/bin/bash
diff --git a/pkg/limayaml/defaults.go b/pkg/limayaml/defaults.go
index a3840cc2..61422b45 100644
--- a/pkg/limayaml/defaults.go
+++ b/pkg/limayaml/defaults.go
@@ -21,6 +21,13 @@ import (
 	"github.com/xorcare/pointer"
 )
 
+const (
+	Default9pSecurityModel   string = "mapped-xattr"
+	Default9pProtocolVersion string = "9p2000.L"
+	Default9pMsize           string = "128KiB"
+	Default9pCache           string = "mmap"
+)
+
 func defaultContainerdArchives() []File {
 	const nerdctlVersion = "0.18.0"
 	location := func(goarch string) string {
@@ -367,6 +374,18 @@ func FillDefault(y, d, o *LimaYAML, filePath string) {
 			if mount.SSHFS.FollowSymlinks != nil {
 				mounts[i].SSHFS.FollowSymlinks = mount.SSHFS.FollowSymlinks
 			}
+			if mount.NineP.SecurityModel != nil {
+				mounts[i].NineP.SecurityModel = mount.NineP.SecurityModel
+			}
+			if mount.NineP.ProtocolVersion != nil {
+				mounts[i].NineP.ProtocolVersion = mount.NineP.ProtocolVersion
+			}
+			if mount.NineP.Msize != nil {
+				mounts[i].NineP.Msize = mount.NineP.Msize
+			}
+			if mount.NineP.Cache != nil {
+				mounts[i].NineP.Cache = mount.NineP.Cache
+			}
 			if mount.Writable != nil {
 				mounts[i].Writable = mount.Writable
 			}
@@ -385,11 +404,33 @@ func FillDefault(y, d, o *LimaYAML, filePath string) {
 		if mount.SSHFS.FollowSymlinks == nil {
 			mount.SSHFS.FollowSymlinks = pointer.Bool(false)
 		}
+		if mount.NineP.SecurityModel == nil {
+			mounts[i].NineP.SecurityModel = pointer.String(Default9pSecurityModel)
+		}
+		if mount.NineP.ProtocolVersion == nil {
+			mounts[i].NineP.ProtocolVersion = pointer.String(Default9pProtocolVersion)
+		}
+		if mount.NineP.Msize == nil {
+			mounts[i].NineP.Msize = pointer.String(Default9pMsize)
+		}
+		if mount.NineP.Cache == nil {
+			mounts[i].NineP.Cache = pointer.String(Default9pCache)
+		}
 		if mount.Writable == nil {
 			mount.Writable = pointer.Bool(false)
 		}
 	}
 
+	if y.MountType == nil {
+		y.MountType = d.MountType
+	}
+	if o.MountType != nil {
+		y.MountType = o.MountType
+	}
+	if y.MountType == nil || *y.MountType == "" {
+		y.MountType = pointer.String(REVSSHFS)
+	}
+
 	// Note: DNS lists are not combined; highest priority setting is picked
 	if len(y.DNS) == 0 {
 		y.DNS = d.DNS
diff --git a/pkg/limayaml/limayaml.go b/pkg/limayaml/limayaml.go
index 47ac646c..67236c2d 100644
--- a/pkg/limayaml/limayaml.go
+++ b/pkg/limayaml/limayaml.go
@@ -14,6 +14,7 @@ type LimaYAML struct {
 	Memory            *string           `yaml:"memory,omitempty" json:"memory,omitempty"` // go-units.RAMInBytes
 	Disk              *string           `yaml:"disk,omitempty" json:"disk,omitempty"`     // go-units.RAMInBytes
 	Mounts            []Mount           `yaml:"mounts,omitempty" json:"mounts,omitempty"`
+	MountType         *MountType        `yaml:"mountType,omitempty" json:"mountType,omitempty"`
 	SSH               SSH               `yaml:"ssh,omitempty" json:"ssh,omitempty"` // REQUIRED (FIXME)
 	Firmware          Firmware          `yaml:"firmware,omitempty" json:"firmware,omitempty"`
 	Video             Video             `yaml:"video,omitempty" json:"video,omitempty"`
@@ -32,10 +33,14 @@ type LimaYAML struct {
 }
 
 type Arch = string
+type MountType = string
 
 const (
 	X8664   Arch = "x86_64"
 	AARCH64 Arch = "aarch64"
+
+	REVSSHFS MountType = "reverse-sshfs"
+	NINEP    MountType = "9p"
 )
 
 type File struct {
@@ -48,6 +53,7 @@ type Mount struct {
 	Location string `yaml:"location" json:"location"` // REQUIRED
 	Writable *bool  `yaml:"writable,omitempty" json:"writable,omitempty"`
 	SSHFS    SSHFS  `yaml:"sshfs,omitempty" json:"sshfs,omitempty"`
+	NineP    NineP  `yaml:"9p,omitempty" json:"9p,omitempty"`
 }
 
 type SSHFS struct {
@@ -55,6 +61,13 @@ type SSHFS struct {
 	FollowSymlinks *bool `yaml:"followSymlinks,omitempty" json:"followSymlinks,omitempty"`
 }
 
+type NineP struct {
+	SecurityModel   *string `yaml:"securityModel,omitempty" json:"securityModel,omitempty"`
+	ProtocolVersion *string `yaml:"protocolVersion,omitempty" json:"protocolVersion,omitempty"`
+	Msize           *string `yaml:"msize,omitempty" json:"msize,omitempty"`
+	Cache           *string `yaml:"cache,omitempty" json:"cache,omitempty"`
+}
+
 type SSH struct {
 	LocalPort *int `yaml:"localPort,omitempty" json:"localPort,omitempty"`
 
diff --git a/pkg/limayaml/validate.go b/pkg/limayaml/validate.go
index 167bb9f8..6c8ada3e 100644
--- a/pkg/limayaml/validate.go
+++ b/pkg/limayaml/validate.go
@@ -102,6 +102,10 @@ func Validate(y LimaYAML, warn bool) error {
 		} else if !st.IsDir() {
 			return fmt.Errorf("field `mounts[%d].location` refers to a non-directory path: %q: %w", i, f.Location, err)
 		}
+
+		if _, err := units.RAMInBytes(*f.NineP.Msize); err != nil {
+			return fmt.Errorf("field `msize` has an invalid value: %w", err)
+		}
 	}
 
 	if *y.SSH.LocalPort != 0 {
@@ -110,6 +114,12 @@ func Validate(y LimaYAML, warn bool) error {
 		}
 	}
 
+	switch *y.MountType {
+	case REVSSHFS, NINEP:
+	default:
+		return fmt.Errorf("field `mountType` must be %q or %q , got %q", REVSSHFS, NINEP, *y.MountType)
+	}
+
 	// y.Firmware.LegacyBIOS is ignored for aarch64, but not a fatal error.
 
 	for i, p := range y.Provision {
diff --git a/pkg/qemu/qemu.go b/pkg/qemu/qemu.go
index a35c7af6..64393fb1 100644
--- a/pkg/qemu/qemu.go
+++ b/pkg/qemu/qemu.go
@@ -16,6 +16,7 @@ import (
 	"github.com/lima-vm/lima/pkg/downloader"
 	"github.com/lima-vm/lima/pkg/iso9660util"
 	"github.com/lima-vm/lima/pkg/limayaml"
+	"github.com/lima-vm/lima/pkg/localpathutil"
 	"github.com/lima-vm/lima/pkg/networks"
 	qemu "github.com/lima-vm/lima/pkg/qemu/const"
 	"github.com/lima-vm/lima/pkg/qemu/imgutil"
@@ -412,7 +413,28 @@ func Cmdline(cfg Config) (string, []string, error) {
 	args = append(args, "-chardev", fmt.Sprintf("socket,id=%s,path=%s,server=on,wait=off,logfile=%s", serialChardev, serialSock, serialLog))
 	args = append(args, "-serial", "chardev:"+serialChardev)
 
-	// We also want to enable vsock and virtfs here, but QEMU does not support vsock and virtfs for macOS hosts
+	// We also want to enable vsock here, but QEMU does not support vsock for macOS hosts
+
+	if *y.MountType == limayaml.NINEP {
+		for i, f := range y.Mounts {
+			tag := fmt.Sprintf("mount%d", i)
+			expanded, err := localpathutil.Expand(f.Location)
+			if err != nil {
+				return "", nil, err
+			}
+			if err := os.MkdirAll(expanded, 0755); err != nil {
+				return "", nil, err
+			}
+			options := "local"
+			options += fmt.Sprintf(",mount_tag=%s", tag)
+			options += fmt.Sprintf(",path=%s", expanded)
+			options += fmt.Sprintf(",security_model=%s", *f.NineP.SecurityModel)
+			if !*f.Writable {
+				options += ",readonly"
+			}
+			args = append(args, "-virtfs", options)
+		}
+	}
 
 	// QMP
 	qmpSock := filepath.Join(cfg.InstanceDir, filenames.QMPSock)


Test patch:
diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
index 89630e26..c3effc97 100644
--- a/.github/workflows/test.yml
+++ b/.github/workflows/test.yml
@@ -115,6 +115,13 @@ jobs:
         retry_on: error
         max_attempts: 3
         command: ./hack/test-example.sh examples/alpine.yaml
+    - name: "Test experimental-9p.yaml"
+      uses: nick-invision/retry@v2
+      with:
+        timeout_minutes: 30
+        retry_on: error
+        max_attempts: 3
+        command: ./hack/test-example.sh examples/experimental-9p.yaml
     - name: "Test vmnet.yaml"
       uses: nick-invision/retry@v2
       with:
diff --git a/pkg/cidata/template_test.go b/pkg/cidata/template_test.go
index 28fd5fe9..27e00248 100644
--- a/pkg/cidata/template_test.go
+++ b/pkg/cidata/template_test.go
@@ -2,6 +2,7 @@ package cidata
 
 import (
 	"io"
+	"strings"
 	"testing"
 
 	"gotest.tools/v3/assert"
@@ -15,10 +16,11 @@ func TestTemplate(t *testing.T) {
 		SSHPubKeys: []string{
 			"ssh-rsa dummy foo@example.com",
 		},
-		Mounts: []string{
-			"/Users/dummy",
-			"/Users/dummy/lima",
+		Mounts: []Mount{
+			{Target: "/Users/dummy"},
+			{Target: "/Users/dummy/lima"},
 		},
+		MountType: "reverse-sshfs",
 	}
 	layout, err := ExecuteTemplate(args)
 	assert.NilError(t, err)
@@ -27,5 +29,37 @@ func TestTemplate(t *testing.T) {
 		b, err := io.ReadAll(f.Reader)
 		assert.NilError(t, err)
 		t.Log(string(b))
+		if f.Path == "user-data" {
+			// mounted later
+			assert.Assert(t, !strings.Contains(string(b), "mounts:"))
+		}
+	}
+}
+
+func TestTemplate9p(t *testing.T) {
+	args := TemplateArgs{
+		Name: "default",
+		User: "foo",
+		UID:  501,
+		SSHPubKeys: []string{
+			"ssh-rsa dummy foo@example.com",
+		},
+		Mounts: []Mount{
+			{Tag: "mount0", Target: "/Users/dummy", Type: "9p", Options: "ro,trans=virtio"},
+			{Tag: "mount1", Target: "/Users/dummy/lima", Type: "9p", Options: "rw,trans=virtio"},
+		},
+		MountType: "9p",
+	}
+	layout, err := ExecuteTemplate(args)
+	assert.NilError(t, err)
+	for _, f := range layout {
+		t.Logf("=== %q ===", f.Path)
+		b, err := io.ReadAll(f.Reader)
+		assert.NilError(t, err)
+		t.Log(string(b))
+		if f.Path == "user-data" {
+			// mounted at boot
+			assert.Assert(t, strings.Contains(string(b), "mounts:"))
+		}
 	}
 }
diff --git a/pkg/limayaml/defaults_test.go b/pkg/limayaml/defaults_test.go
index 17cbfb0b..cdc800b5 100644
--- a/pkg/limayaml/defaults_test.go
+++ b/pkg/limayaml/defaults_test.go
@@ -101,6 +101,7 @@ func TestFillDefault(t *testing.T) {
 		Mounts: []Mount{
 			{Location: "/tmp"},
 		},
+		MountType: pointer.String(NINEP),
 		Provision: []Provision{
 			{Script: "#!/bin/true"},
 		},
@@ -136,8 +137,14 @@ func TestFillDefault(t *testing.T) {
 	expect.Mounts[0].Writable = pointer.Bool(false)
 	expect.Mounts[0].SSHFS.Cache = pointer.Bool(true)
 	expect.Mounts[0].SSHFS.FollowSymlinks = pointer.Bool(false)
+	expect.Mounts[0].NineP.SecurityModel = pointer.String(Default9pSecurityModel)
+	expect.Mounts[0].NineP.ProtocolVersion = pointer.String(Default9pProtocolVersion)
+	expect.Mounts[0].NineP.Msize = pointer.String(Default9pMsize)
+	expect.Mounts[0].NineP.Cache = pointer.String(Default9pCache)
 	// Only missing Mounts field is Writable, and the default value is also the null value: false
 
+	expect.MountType = pointer.String(NINEP)
+
 	expect.Provision = y.Provision
 	expect.Provision[0].Mode = ProvisionModeSystem
 
@@ -267,9 +274,14 @@ func TestFillDefault(t *testing.T) {
 	expect.Containerd.Archives[0].Arch = *d.Arch
 	expect.Mounts[0].SSHFS.Cache = pointer.Bool(true)
 	expect.Mounts[0].SSHFS.FollowSymlinks = pointer.Bool(false)
+	expect.Mounts[0].NineP.SecurityModel = pointer.String(Default9pSecurityModel)
+	expect.Mounts[0].NineP.ProtocolVersion = pointer.String(Default9pProtocolVersion)
+	expect.Mounts[0].NineP.Msize = pointer.String(Default9pMsize)
+	expect.Mounts[0].NineP.Cache = pointer.String(Default9pCache)
 	expect.HostResolver.Hosts = map[string]string{
 		"default.": d.HostResolver.Hosts["default"],
 	}
+	expect.MountType = pointer.String(REVSSHFS)
 
 	y = LimaYAML{}
 	FillDefault(&y, &d, &LimaYAML{}, filePath)
@@ -353,6 +365,12 @@ func TestFillDefault(t *testing.T) {
 					Cache:          pointer.Bool(false),
 					FollowSymlinks: pointer.Bool(true),
 				},
+				NineP: NineP{
+					SecurityModel:   pointer.String("mapped-file"),
+					ProtocolVersion: pointer.String("9p2000"),
+					Msize:           pointer.String("8KiB"),
+					Cache:           pointer.String("none"),
+				},
 			},
 		},
 		Provision: []Provision{
@@ -414,6 +432,12 @@ func TestFillDefault(t *testing.T) {
 	expect.Mounts[0].Writable = pointer.Bool(true)
 	expect.Mounts[0].SSHFS.Cache = pointer.Bool(false)
 	expect.Mounts[0].SSHFS.FollowSymlinks = pointer.Bool(true)
+	expect.Mounts[0].NineP.SecurityModel = pointer.String("mapped-file")
+	expect.Mounts[0].NineP.ProtocolVersion = pointer.String("9p2000")
+	expect.Mounts[0].NineP.Msize = pointer.String("8KiB")
+	expect.Mounts[0].NineP.Cache = pointer.String("none")
+
+	expect.MountType = pointer.String(NINEP)
 
 	// o.Networks[1] is overriding the d.Networks[0].Lima entry for the "def0" interface
 	expect.Networks = append(append(d.Networks, y.Networks...), o.Networks[0])