Repository: moby/moby (language: go)

Issue:
Driver "zfs" failed to remove root filesystem: "cannot open '<dataset-path>': dataset does not exist"
<!--
If you are reporting a new issue, make sure that we do not have any duplicates
already open. You can ensure this by searching the issue list for this
repository. If there is a duplicate, please close your issue and add a comment
to the existing issue instead.

If you suspect your issue is a bug, please edit your issue description to
include the BUG REPORT INFORMATION shown below. If you fail to provide this
information within 7 days, we cannot debug your issue and will close it. We
will, however, reopen it if you later provide the information.

For more information about reporting issues, see
https://github.com/moby/moby/blob/master/CONTRIBUTING.md#reporting-other-issues

---------------------------------------------------
GENERAL SUPPORT INFORMATION
---------------------------------------------------

The GitHub issue tracker is for bug reports and feature requests.
General support for **docker** can be found at the following locations:

- Docker Support Forums - https://forums.docker.com
- Slack - community.docker.com #general channel
- Post a question on StackOverflow, using the Docker tag

General support for **moby** can be found at the following locations:

- Moby Project Forums - https://forums.mobyproject.org
- Slack - community.docker.com #moby-project channel
- Post a question on StackOverflow, using the Moby tag

---------------------------------------------------
BUG REPORT INFORMATION
---------------------------------------------------
Use the commands below to provide key information from your environment:
You do NOT have to include this information if this is a FEATURE REQUEST
-->

**Description**
we are using [openebs/zfs ](https://github.com/openebs/zfs-localpv/blob/develop/deploy/zfs-operator.yaml) and we have multiple reports where users are having issues that boils down to the docker with zfs storage driver.

<!--
Briefly describe the problem you are having in a few paragraphs.
-->
Pods are stuck in terminating state:
```
container <ID> driver "zfs" failed to remove root filesystem: exit status 1: "zfs fs destroy -r <dataset-path>" => cannot open '<dataset-path>': dataset does not exist"
```

Reports:
- https://jira.ixsystems.com/browse/NAS-112418
- https://www.truenas.com/community/threads/openebs-zfs-driver-removal-in-progress-for-4-weeks.96192/
- https://help.nextcloud.com/t/solved-cannot-start-service-db-error-creating-zfs-mount-after-zfs-snapshot-rollback/61695

This seems to have a workaround where you manually create missing datasets to make docker happy and things come back to normal.. which is tedious.

**Steps to reproduce the issue:**
Unable to find concrete steps to reproduce, it seems to be a raise condition or non-atomic operation where driver cloudn't finish the removal of the layers datasets but gets into crashloop and most of times its on system restart / power cut when pods are stuck in terminating state giving the following logs:

**Describe the results you received:**

```bash
truenas# docker ps -a
CONTAINER ID   IMAGE                         COMMAND                  CREATED       STATUS                     PORTS     NAMES
d39f9d831680   openebs/zfs-driver            "/usr/local/bin/zfs-…"   4 weeks ago   Removal In Progress                  k8s_openebs-zfs-plugin_openebs-zfs-node-ds5sd_kube-system_d692ac39-ac25-4da6-adbb-93615d9cef3b_203
4f7053116229   rancher/pause:3.1             "/pause"                 4 weeks ago   Exited (255) 4 weeks ago             k8s_POD_openebs-zfs-node-ds5sd_kube-system_d692ac39-ac25-4da6-adbb-93615d9cef3b_27


truenas# k get pods -A
NAMESPACE          NAME                                    READY   STATUS        RESTARTS   AGE
kube-system        openebs-zfs-node-ds5sd                  0/2     Terminating   315        106d
kube-system        coredns-7448499f4d-hmsv9                1/1     Running       0          8h
kube-system        openebs-zfs-controller-0                5/5     Running       0          8h

#######
## LOGS:
#######
Oct 27 01:36:10 truenas k3s[8629]: I1027 01:36:10.703743    8629 scope.go:111] "RemoveContainer" containerID="d39f9d83168090d640c8473d2311ec52be722e58951549aef32f187e59f90f8b"
Oct 27 01:36:10 truenas dockerd[8071]: time="2021-10-27T01:36:10.739172443+02:00" level=error msg="Error removing mounted layer d39f9d83168090d640c8473d2311ec52be722e58951549aef32f187e59f90f8b: exit status 1: \"/usr/sbin/zfs fs destroy -r storage_404/ix-applications/docker/3114cdddca0eb01f1de848925eba4d027796b424b91f12d843673e174667bd24\" => cannot open 'storage_404/ix-applications/docker/3114cdddca0eb01f1de848925eba4d027796b424b91f12d843673e174667bd24': dataset does not exist\n"
Oct 27 01:36:10 truenas dockerd[8071]: time="2021-10-27T01:36:10.739870694+02:00" level=error msg="Handler for DELETE /v1.41/containers/d39f9d83168090d640c8473d2311ec52be722e58951549aef32f187e59f90f8b returned error: container d39f9d83168090d640c8473d2311ec52be722e58951549aef32f187e59f90f8b: driver \"zfs\" failed to remove root filesystem: exit status 1: \"/usr/sbin/zfs fs destroy -r storage_404/ix-applications/docker/3114cdddca0eb01f1de848925eba4d027796b424b91f12d843673e174667bd24\" => cannot open 'storage_404/ix-applications/docker/3114cdddca0eb01f1de848925eba4d027796b424b91f12d843673e174667bd24': dataset does not exist\n"
Oct 27 01:36:10 truenas k3s[8629]: E1027 01:36:10.740889    8629 remote_runtime.go:296] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to remove container \"d39f9d83168090d640c8473d2311ec52be722e58951549aef32f187e59f90f8b\": Error response from daemon: container d39f9d83168090d640c8473d2311ec52be722e58951549aef32f187e59f90f8b: driver \"zfs\" failed to remove root filesystem: exit status 1: \"/usr/sbin/zfs fs destroy -r storage_404/ix-applications/docker/3114cdddca0eb01f1de848925eba4d027796b424b91f12d843673e174667bd24\" => cannot open 'storage_404/ix-applications/docker/3114cdddca0eb01f1de848925eba4d027796b424b91f12d843673e174667bd24': dataset does not exist" containerID="d39f9d83168090d640c8473d2311ec52be722e58951549aef32f187e59f90f8b"
Oct 27 01:36:10 truenas k3s[8629]: E1027 01:36:10.741271    8629 kuberuntime_gc.go:146] "Failed to remove container" err="rpc error: code = Unknown desc = failed to remove container \"d39f9d83168090d640c8473d2311ec52be722e58951549aef32f187e59f90f8b\": Error response from daemon: container d39f9d83168090d640c8473d2311ec52be722e58951549aef32f187e59f90f8b: driver \"zfs\" failed to remove root filesystem: exit status 1: \"/usr/sbin/zfs fs destroy -r storage_404/ix-applications/docker/3114cdddca0eb01f1de848925eba4d027796b424b91f12d843673e174667bd24\" => cannot open 'storage_404/ix-applications/docker/3114cdddca0eb01f1de848925eba4d027796b424b91f12d843673e174667bd24': dataset does not exist" containerID="d39f9d83168090d640c8473d2311ec52be722e58951549aef32f187e59f90f8b"
```

**Describe the results you expected:**


**Additional information you deem important (e.g. issue happens only occasionally):**

**Output of `docker version`:**
```
Client: Docker Engine - Community
 Version:           20.10.11
 API version:       1.41
 Go version:        go1.16.9
 Git commit:        dea9396
 Built:             Thu Nov 18 00:37:22 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.11
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.9
  Git commit:       847da18
  Built:            Thu Nov 18 00:35:31 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.12
  GitCommit:        7b11cfaabd73bb80907dd23182b9347b4245eb5d
 runc:
  Version:          1.0.2
  GitCommit:        v1.0.2-0-g52b36a2
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
```

**Output of `docker info`:**

```
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.6.3-docker)
  scan: Docker Scan (Docker Inc., v0.9.0)

Server:
 Containers: 36
  Running: 33
  Paused: 0
  Stopped: 3
 Images: 23
 Server Version: 20.10.11
 Storage Driver: zfs
  Zpool: evo
  Zpool Health: ONLINE
  Parent Dataset: evo/ix-applications/docker
  Space Used By Parent: 2543288320
  Space Available: 16198967296
  Parent Quota: no
  Compression: lz4
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7b11cfaabd73bb80907dd23182b9347b4245eb5d
 runc version: v1.0.2-0-g52b36a2
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.10.70+truenas
 Operating System: Debian GNU/Linux 11 (bullseye)
 OSType: linux
 Architecture: x86_64
 CPUs: 6
 Total Memory: 13.53GiB
 Name: truenas.local
 ID: NDIM:5V6V:MQ5E:QNFP:VPAB:4EYO:YKNL:EF55:WW5R:2IQM:ARAF:5NIG
 Docker Root Dir: /mnt/evo/ix-applications/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
```

**Additional environment details (AWS, VirtualBox, physical, etc.):**



Discussion / hints:
The `dataset does not exist` message is also mentioned in this thread: https://github.com/moby/moby/issues/40132
Hi have the same problem.

Can this case, where the ZFS volume to be deleted does not exists, be handled as a warning in the logs instead of a failure ?

## The details

A container is dead
```sh
$ docker ps -a | grep quizzical_kapitsa
CONTAINER ID   IMAGE                                      COMMAND                  CREATED             STATUS         PORTS               NAMES
e5d845b7f04d   7e652150d4f5                               "/bin/sh -c './confi…"   22 months ago       Dead                               quizzical_kapitsa
```

The creation date is strange as it was created by a **docker-compose build** a few minutes before.

Let's remove it
```sh
$ docker rm quizzical_kapitsa
Error response from daemon: container e5d845b7f04da595028ed461b9ee7ef1de351a2dcf3d55eab3fe35f89bbb05a4: driver "zfs" failed to remove root filesystem: exit status 1: "/usr/sbin/zfs destroy -r data0/docker/fcd1043c9b5b5e71f3afb7cf69e3a8efaaf31ba8b8a0b925fec317ac94a40445" => cannot open 'data0/docker/fcd1043c9b5b5e71f3afb7cf69e3a8efaaf31ba8b8a0b925fec317ac94a40445': dataset does not exist
```

The error from the docker daemon
```text
Sep 17 14:32:58 iota dockerd[26393]: time="2024-09-17T14:32:58.358043884Z" level=error msg="Handler for DELETE /v1.47/containers/quizzical_kapitsa returned error: container e5d845b7f04da595028ed461b9ee7ef1de351a2dcf3d55eab3fe35f89bbb05a4: driver \"zfs\" failed to remove root filesystem: exit status 1: \"/usr/sbin/zfs destroy -r data0/docker/fcd1043c9b5b5e71f3afb7cf69e3a8efaaf31ba8b8a0b925fec317ac94a40445\" => cannot open 'data0/docker/fcd1043c9b5b5e71f3afb7cf69e3a8efaaf31ba8b8a0b925fec317ac94a40445': dataset does not exist\n"
```

Let's recreate the ZFS volumes so docker can destroy them
```sh
root@iota:~# zfs create data0/docker/fcd1043c9b5b5e71f3afb7cf69e3a8efaaf31ba8b8a0b925fec317ac94a40445-init
root@iota:~# zfs create data0/docker/fcd1043c9b5b5e71f3afb7cf69e3a8efaaf31ba8b8a0b925fec317ac94a40445
```

And it worked
```sh
$ docker rm quizzical_kapitsa
quizzical_kapitsa
```

## Versions

Docker version

```text
Client: Docker Engine - Community
 Version:           27.2.1
 API version:       1.47
 Go version:        go1.22.7
 Git commit:        9e34c9b
 Built:             Fri Sep  6 12:08:06 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          27.2.1
  API version:      1.47 (minimum version 1.24)
  Go version:       go1.22.7
  Git commit:       8b539b8
  Built:            Fri Sep  6 12:08:06 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.7.22
  GitCommit:        7f7fdf5fed64eb6a7caf99b3e12efcf9d60e311c
 runc:
  Version:          1.1.14
  GitCommit:        v1.1.14-0-g2c9f560
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
```

Kernel version: Linux version 6.10.9-amd64 (debian-kernel@lists.debian.org) (x86_64-linux-gnu-gcc-13 (Debian 13.3.0-6) 13.3.0, GNU ld (GNU Binutils for Debian) 2.43.1) #1 SMP PREEMPT_DYNAMIC Debian 6.10.9-1 (2024-09-08)

ZFS version: 2.2.6-1




Fix patch:
diff --git a/daemon/graphdriver/zfs/zfs.go b/daemon/graphdriver/zfs/zfs.go
index cca34d6284739..625daf43786b3 100644
--- a/daemon/graphdriver/zfs/zfs.go
+++ b/daemon/graphdriver/zfs/zfs.go
@@ -361,6 +361,17 @@ func (d *Driver) Remove(id string) error {
 	name := d.zfsPath(id)
 	dataset := zfs.Dataset{Name: name}
 	err := dataset.Destroy(zfs.DestroyRecursive)
+	if err != nil {
+		var errZfs *zfs.Error
+		isZfsError := errors.As(err, &errZfs)
+		if isZfsError && strings.HasSuffix(strings.TrimSpace(errZfs.Stderr), "dataset does not exist") {
+			log.G(context.TODO()).WithFields(log.Fields{
+				"error":          err,
+				"storage-driver": "zfs",
+			}).Warnf("Tried to destroy inexistent dataset %q", name)
+			err = nil
+		}
+	}
 	if err == nil {
 		d.Lock()
 		delete(d.filesystemsCache, name)


Test patch: