Repository: lima-vm/lima

Issue:
docs: discourage `/tmp/lima` on shared hosts (and deprecate it?)
Using `/tmp/lima` is discouraged on shared hosts, as another user on the host can peek/poke the directory.

Maybe we should deprecate and remove it in v2.0 ?
A user can still enable the directory by specifying `--mount /tmp/lima:w`.

Discussion:
@jandubois: > Using `/tmp/lima` is discouraged on shared hosts, as another user on the host can peek/poke the directory.

Only users with the same GID can poke because we create the directory with `750` permissions. Maybe we should create non-existing directories with `700` instead?

Since `/tmp` normally has the sticky bit set, only the owner or `root` can change the permission flags, even if the user otherwise has write access to the directory itself.@jandubois: SSH agent forwarding puts the agent socket in a directory on `/tmp`, relying on file mode protection bits being sticky. If it is good enough for that use case, then it should be good enough for `/tmp/lima`.@AkihiroSuda: > Maybe we should create non-existing directories with 700 instead?

Yes, but TOCTOU still has to be cared carefully

BTW, in the current design multiple users cannot run Lima simultaneously by default, so it should be renamed to `/tmp/lima-{{.Username}}`?

@jandubois: > BTW, in the current design multiple users cannot run Lima simultaneously by default, so it should be renamed to `/tmp/lima-{{.Username}}`?

We currently use `{{.GlobalTempDir}}` in our default mounts:

```yaml
mounts:
- location: "{{.GlobalTempDir}}/lima"
  mountPoint: /tmp/lima
  writable: true
```

We talked about using `{{.TempDir}}` whenever we are willing to make an incompatible change (Lima 2.0 release). It will use the value of `$TMPDIR` instead, so should be a private directory.

We could add additional code that checks if the return value is still `/tmp`, and if it is, we check if `/run/user/$UID` exists, and use that if it does.

It would only affect Linux, as macOS always has `$TMPDIR` pointing to a private temp directory.

Not sure I like it though, a bit too much magic, but what do you think?@jandubois: > Maybe we should create non-existing directories with `700` instead?

It looks like this is currently not very consistent. We seem to create directories for forwarded sockets with 750 permissions. Mount locations are created with 755 when using reverse-sshfs and 750 when using virtiofs.

I would suggest to change all of them to 700. This only applies to directories created by Lima. If they already exist, then their existing permissions should not be changed.@afbjorklund: This was one of the reasons for making /tmp/lima into a variable:

* https://github.com/lima-vm/lima/issues/2339

The other reason was that Fedora puts /tmp in RAM so the space is limited...

If you are doing something like building images, might want to use /var/tmp@thomasjm: > BTW, in the current design multiple users cannot run Lima simultaneously by default, so it should be renamed to /tmp/lima-{{.Username}} ?

> We talked about using {{.TempDir}} whenever we are willing to make an incompatible change (Lima 2.0 release). It will use the value of $TMPDIR instead, so should be a private directory.

Either of these ideas would improve the story for multiple users on the same machine using Lima. But both of them would still cause different Lima instances belonging to the same user (or login session) to share the same `/tmp` dir. It doesn't make sense to me why separate VMs would share a `/tmp` dir.

Possible dumb question, but why do the default Lima templates mount "/tmp" from the host at all? It seems like the cleanest option would be to not do this.@AkihiroSuda: > $TMPDIR

The path is long, ugly, and hard to locate from a terminal

> Possible dumb question, but why do the default Lima templates mount "/tmp" from the host at all? It seems like the cleanest option would be to not do this.

Because I wanted to have a writable directory while keeping the home read-only by default
@thomasjm: > Because I wanted to have a writable directory while keeping the home read-only by default

Gotcha -- is that something that's important for the operation of Lima somehow (or some of the templates)?

To me it's somewhat surprising as a user that Lima auto-mounts anything at all, and I always have to go digging through the docs to remind myself how it works. As far as the docs go, it's not mentioned on the [filesystem mounts](https://lima-vm.io/docs/config/mount/) page; the only reference I can find is under [examples](https://lima-vm.io/docs/examples/).@jandubois: > Because I wanted to have a writable directory while keeping the home read-only by default

I always mount the home directory writable, so I rarely ever need the temp mount.

I think we should consider making it opt-in instead of opt-out in Lima 2.0.

A simple way to do this would be to move it to `template://_default/temp-mount.yaml`.

I've already been contemplating if we should allow specifying additional base templates on the commandline (they would be relative to the first template if using a relative path), so you could write:

```shell
limactl create -y template://ubuntu _default/temp-mount.yaml
```

and it would be the same as if the template was

```yaml
base:
- ubuntu.yaml
- _default/mounts.yaml
- _default/temp-mount.yaml
```

Of course this is still somewhat clunky, and I'm hoping that we could use something like `--param temp-mounts` instead, but that depends on how soon we want to have a Lima 2.0 release.

Or we could add a `default://` URL scheme, so `default://temp-mount` would be shorthand for `template://_default/temp-mount`. It also hides the somewhat ugly implementation details of the `_default` subdirectory from the user:

```shell
limactl create -y template://ubuntu default://temp-mount
```

Anyways, lots of bikesheds that can be painted...@thomasjm: > I've already been contemplating if we should allow specifying additional base templates on the commandline

I like the idea of increasing the configurability. But doing it in terms of templates would require the user to dig into and understand how the templates work. Which to me would mean either looking in the Lima repository or finding the templates dir where it's installed on my system, neither of which is too appealing.

Lima already has the `--mount` argument, which IMO is the right level of granularity for this. Making it even more similar to Docker's `--volume` argument for bind-mounting would be even better :)

I mean, consider the clarity of something like this (inventing some syntax a bit):

```
limactl create -y template://ubuntu --mount "$(mktemp -d):/tmp:w"
```@jandubois: > Which to me would mean either looking in the Lima repository or finding the templates dir where it's installed on my system

Regardless of this current discussion, I think the documentation for templates need some serious improvements. So at least for all bundled templates it needs to contain the information how they can be further customized / parameterized.

> I mean, consider the clarity of something like this (inventing some syntax a bit):
> 
> ```
> limactl create -y template://ubuntu --mount "$(mktemp -d):/tmp:w"
> ```

I like it, and should be easy to implement.@AkihiroSuda: > limactl create -y template://ubuntu --mount "$(mktemp -d):/tmp:w"

This already works
@afbjorklund: > To me it's somewhat surprising as a user that Lima auto-mounts anything at all, and I always have to go digging through the docs to remind myself how it works.

This should at least be mentioned, not all users are coming from a Docker Machine and expecting all of their home directory to be exported to the VM (with the `/tmp/lima` being the writable workaround for the read-only default mount in 1.0)

For instance we have some ex-Vagrant users, that are expecting the VM to use a unique user name (like "vagrant" or "lima") and only export the current working directory ($PWD) under some unique location (like "/vagrant") and not the home.

----
There were also some similar "complaints" about containerd being installed by default, but not hard to avoid that either.

`--containerd none`@jandubois: > > limactl create -y template://ubuntu --mount "$(mktemp -d):/tmp:w"
> 
> This already works

I don't think so. `--mount /foo:w` works, but `--mount /foo:/bar:w` doesn't. You can specify the `location`, but not the `mountPoint`. Is easy to fix though: if the argument has 2 `:` in it, then the middle term is the `mountPoint`.@jandubois: > For instance we have some ex-Vagrant users, that are expecting the VM to use a unique user name (like "vagrant" or "lima") and only export the current working directory ($PWD) under some unique location (like "/vagrant") and not the home.

Not every use case can be the default, unless they are all the same use case.  The history of the Lima project is to make it easy to run containers on macOS and pretend they are running locally and not in a VM.

We have worked quite a bit to make Lima more generally useful and configurable, but that doesn't mean we have to abandon the original use case. Note that "make it easy" was not accidental, but a design goal.

You can always use `--plain` if you don't want any of the Lima features that integrate the VM with the host.  Or you can easily create a template for whatever you want now:

```yaml
base: template://_images/ubuntu

containerd:
  system: false
  user: false

user:
  name: vagrant
  uid: 1000
  home: /home/vagrant
```

We do not support referencing `$PWD` or other environment variables in the template. It is not clear how they should work: do they need to be interpolated when you create the instance, or when you start it. Should `limactl restart` change the mount locations because you are in a different directory now?
@msimkunas: Slightly off-topic but why is the user's home directory automounted? I was a little taken aback by this. I'd rather Lima used a secure-out-of-the-box approach and would not expose the contents of $HOME to the VM by default.

Of course, it's the user's fault if they run an insecure template but the default configuration should at the very least attempt to contain the potential damage. Exposing the entirety of $HOME to the VM seems completely unnecessary. 

In most cases, this may not even be desired because you will usually want to mount a directory that contains your projects only anyway.@AkihiroSuda: > Slightly off-topic but why is the user's home directory automounted? 

For allowing `(docker|podman|nerdctl) run -v $(pwd):/mnt` to just work, as in Docker Desktop.

> I'd rather Lima used a secure-out-of-the-box approach and would not expose the contents of $HOME to the VM by default.

You can use `limactl start --mount-none` to disable all the mounts.

> Exposing the entirety of $HOME to the VM seems completely unnecessary.

We could intercept invocation of `(docker|podman|nerdctl) run -v` and make mounts dynamically, but it could be quite complicated and less robust.

> In most cases, this may not even be desired because you will usually want to mount a directory that contains your projects only anyway.

Lima v2.0 is planned to have `limactl shell --sync-host-workdir` to implement that (but rsync instead of mount, probably)
- https://github.com/lima-vm/lima/issues/3711

```console
$ limactl start --mount-none template://claude
$ cd ~/some-project
$ limactl shell --sync-host-workdir claude claude
Changed files:
- foo.go
- Makefile

⚠️Accept the changes?
→ Yes
  No
  View the changed contents
```
@msimkunas: @AkihiroSuda Thanks for clearing it up. I've also found that you can basically override everything via a template, so that works as well.@afbjorklund: > Lima already has the `--mount` argument, which IMO is the right level of granularity for this. Making it even more similar to Docker's `--volume` argument for bind-mounting would be even better :)

I used the same option (`--volume`) for Podman Machine, and it has been confusing users ever since.
For some reason they mix it up with Podman Engine volumes and bind mounts, which are different...

So I think having two different names for it is probably a good idea. Even if it's two different projects.
But the mountpoint syntax should be possible to add, if ignoring the name of the flag for a second.

Solution patch:
diff --git a/pkg/driver/vz/vm_darwin.go b/pkg/driver/vz/vm_darwin.go
index 342f73fe..baac69e2 100644
--- a/pkg/driver/vz/vm_darwin.go
+++ b/pkg/driver/vz/vm_darwin.go
@@ -53,7 +53,7 @@ type virtualMachineWrapper struct {
 var vmNetworkFiles = make([]*os.File, 1)
 
 func startVM(ctx context.Context, inst *limatype.Instance, sshLocalPort int) (*virtualMachineWrapper, chan error, error) {
-	usernetClient, err := startUsernet(ctx, inst)
+	usernetClient, stopUsernet, err := startUsernet(ctx, inst)
 	if err != nil {
 		return nil, nil, err
 	}
@@ -118,6 +118,7 @@ func startVM(ctx context.Context, inst *limatype.Instance, sshLocalPort int) (*v
 					wrapper.stopped = true
 					wrapper.mu.Unlock()
 					_ = usernetClient.UnExposeSSH(inst.SSHLocalPort)
+					stopUsernet()
 					errCh <- errors.New("vz driver state stopped")
 				default:
 					logrus.Debugf("[VZ] - vm state change: %q", newState)
@@ -129,22 +130,23 @@ func startVM(ctx context.Context, inst *limatype.Instance, sshLocalPort int) (*v
 	return wrapper, errCh, err
 }
 
-func startUsernet(ctx context.Context, inst *limatype.Instance) (*usernet.Client, error) {
+func startUsernet(ctx context.Context, inst *limatype.Instance) (*usernet.Client, context.CancelFunc, error) {
 	if firstUsernetIndex := limayaml.FirstUsernetIndex(inst.Config); firstUsernetIndex != -1 {
 		nwName := inst.Config.Networks[firstUsernetIndex].Lima
-		return usernet.NewClientByName(nwName), nil
+		return usernet.NewClientByName(nwName), nil, nil
 	}
 	// Start a in-process gvisor-tap-vsock
 	endpointSock, err := usernet.SockWithDirectory(inst.Dir, "", usernet.EndpointSock)
 	if err != nil {
-		return nil, err
+		return nil, nil, err
 	}
 	vzSock, err := usernet.SockWithDirectory(inst.Dir, "", usernet.FDSock)
 	if err != nil {
-		return nil, err
+		return nil, nil, err
 	}
 	os.RemoveAll(endpointSock)
 	os.RemoveAll(vzSock)
+	ctx, cancel := context.WithCancel(ctx)
 	err = usernet.StartGVisorNetstack(ctx, &usernet.GVisorNetstackOpts{
 		MTU:      1500,
 		Endpoint: endpointSock,
@@ -156,10 +158,11 @@ func startUsernet(ctx context.Context, inst *limatype.Instance) (*usernet.Client
 		Subnet: networks.SlirpNetwork,
 	})
 	if err != nil {
-		return nil, err
+		defer cancel()
+		return nil, nil, err
 	}
 	subnetIP, _, err := net.ParseCIDR(networks.SlirpNetwork)
-	return usernet.NewClient(endpointSock, subnetIP), err
+	return usernet.NewClient(endpointSock, subnetIP), cancel, err
 }
 
 func createVM(ctx context.Context, inst *limatype.Instance) (*vz.VirtualMachine, error) {
diff --git a/templates/_default/mounts.yaml b/templates/_default/mounts.yaml
index caaf004e..b934c939 100644
--- a/templates/_default/mounts.yaml
+++ b/templates/_default/mounts.yaml
@@ -1,6 +1,7 @@
 mounts:
 - location: "~"
 
-- location: "{{.GlobalTempDir}}/lima"
-  mountPoint: /tmp/lima
-  writable: true
+# # /tmp/lima is no longer mounted by default since Lima v2.0.
+# - location: "{{.GlobalTempDir}}/lima"
+#  mountPoint: /tmp/lima
+#  writable: true
diff --git a/templates/default.yaml b/templates/default.yaml
index ebf588b9..d1a00616 100644
--- a/templates/default.yaml
+++ b/templates/default.yaml
@@ -38,7 +38,8 @@ disk: null
 # {{.GlobalTempDir}}, and {{.TempDir}}. The global temp dir is always "/tmp" on Unix.
 # "mountPoint" can use these template variables: {{.Home}}, {{.Name}}, {{.Hostname}}, {{.UID}}, {{.User}}, and {{.Param.Key}}.
 # 🟢 Builtin default: [] (Mount nothing)
-# 🔵 This file: Mount the home as read-only, /tmp/lima as writable  (inherited via the `base` mechanism later in this file)
+# 🔵 This file: Mount the home as read-only (inherited via the `base` mechanism later in this file)
+#    Until Lima v1.2, /tmp/lima was mounted too as writable.
 mounts: []
 # - location: "~"
 #  # Configure the mountPoint inside the guest.
@@ -79,10 +80,6 @@ mounts: []
 #    # See https://www.kernel.org/doc/Documentation/filesystems/9p.txt
 #    # 🟢 Builtin default: "fscache" for non-writable mounts, "mmap" for writable mounts
 #    cache: null
-# - location: "/tmp/lima"
-#  # 🟢 Builtin default: false
-#  # 🔵 This file: true (only for "/tmp/lima")
-#  writable: true
 
 # List of mount types not supported by the kernel of this distro.
 # Also used to resolve the default mount type when not explicitly specified.
diff --git a/website/content/en/docs/config/_index.md b/website/content/en/docs/config/_index.md
index 097fca11..a9dd680e 100644
--- a/website/content/en/docs/config/_index.md
+++ b/website/content/en/docs/config/_index.md
@@ -10,7 +10,7 @@ The current default spec:
 - CPU: 4 cores
 - Memory: 4 GiB
 - Disk: 100 GiB
-- Mounts: `~` (read-only), `/tmp/lima` (writable)
+- Mounts: `~` (read-only), `/tmp/lima` (writable; removed in Lima v2.0)
 - SSH: 127.0.0.1:<Random port>
 
 For environment variables, see [Environment Variables](./environment-variables/).
diff --git a/website/content/en/docs/dev/drivers.md b/website/content/en/docs/dev/drivers.md
index bc5d293c..d69d5d43 100644
--- a/website/content/en/docs/dev/drivers.md
+++ b/website/content/en/docs/dev/drivers.md
@@ -3,8 +3,8 @@ title: Virtual Machine Drivers
 weight: 15
 ---
 
- | ⚡ Requirement | Lima >= 0.16.0 | 
- |-------------------|----------------| 
+ | ⚡ Requirement | Lima >= 2.0 |
+ |----------------|-------------|
 
 Lima supports two types of drivers: **internal** and **external**. This architecture allows for extensibility and platform-specific implementations. Drivers are unware whether they are internal or external.
 
diff --git a/website/content/en/docs/examples/_index.md b/website/content/en/docs/examples/_index.md
index 010d3614..6370ffa2 100644
--- a/website/content/en/docs/examples/_index.md
+++ b/website/content/en/docs/examples/_index.md
@@ -10,7 +10,7 @@ lima uname -a
 
 ## Accessing host files
 
-By default, the VM has read-only accesses to `/Users/<USERNAME>` and read-write accesses to `/tmp/lima`.
+By default, the VM has read-only accesses to `/Users/<USERNAME>`.
 
 To allow writing to `/Users/<USERNAME>`:
 ```bash
@@ -20,6 +20,14 @@ limactl edit --mount-writable --mount-type=virtiofs
 Specifying `--mount-type=virtiofs` is not necessary here, but it is highly recommended
 for the best performance and stability.
 
+{{% alert title="Hint" color=success %}}
+Lima prior to v2.0 mounts `/tmp/lima` too in read-write mode.
+
+This directory is no longer mounted by default since Lima v2.0.
+To mount `/tmp/lima` in Lima v2.0 and later, set `--mount /tmp/lima:w`.
+The `:w` suffix indicates read-write mode.
+{{% /alert %}}
+
 ## Running containers
 {{< tabpane text=true >}}
 



Test patch:
diff --git a/pkg/yqutil/yqutil_test.go b/pkg/yqutil/yqutil_test.go
index e1785cad..7fdc5da9 100644
--- a/pkg/yqutil/yqutil_test.go
+++ b/pkg/yqutil/yqutil_test.go
@@ -67,7 +67,7 @@ func TestEvaluateExpressionComplex(t *testing.T) {
 	content := `
 # Expose host directories to the guest, the mount point might be accessible from all UIDs in the guest
 # 🟢 Builtin default: null (Mount nothing)
-# 🔵 This file: Mount the home as read-only, /tmp/lima as writable
+# 🔵 This file: Mount the home as read-only
 mounts:
 - location: "~"
   # Configure the mountPoint inside the guest.
@@ -80,7 +80,7 @@ mounts:
 	expected := `
 # Expose host directories to the guest, the mount point might be accessible from all UIDs in the guest
 # 🟢 Builtin default: null (Mount nothing)
-# 🔵 This file: Mount the home as read-only, /tmp/lima as writable
+# 🔵 This file: Mount the home as read-only
 mounts:
 - location: "~"
   # Configure the mountPoint inside the guest.