Change:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Kubelet v1.19.0 doesn't start on nodes with CRI-O 1.17.2 (storagedriver="btrfs") with BTRFS as root filesystem. Related/previous issues: This error shows up in the journal: The is referring to the btrfs root filesystem, mounted at the time. : : What you expected to happen: Kubelet starts and detects the filesystem backing , even if it is BTRFS. How to reproduce it (as minimally and precisely as possible): Install kubeadm/kubectl 1.19.0 Install CRI-O, I used 1.17.2 from: Configure CRI-O to use Perform "kubeadm init" creating a new cluster Observe journal (check for kubelet issues) Anything else we need to know?: A possible workaround is to make sure a bind mount exists which allows kubelet's logic to find the backing fileystem. Eg. add the following fstab entry and then perform : The resulting mount should look somewhat like this: The workaround has been tested and seems to make kubelet work as expected. Environment: Kubernetes version: 1.19.0 OS: Debian Bullseye (Testing) Kernel: Linux 5.8.5 Others: The package has been installed
/sig node
cc xref
I am encounter the same issue. thank you for your workaround. It seems to work.
I got the same issue with latest stable Docker on Debian (btrfs) and k8s v1.19.2.
I confirm the same issue shows up with on btrfs (on LVM on LUKS, Debian buster), running k3s. No btrfs subvols. The workaround works for me; for k3s I also need to bind mount . I did not experience this issue with 1.18.
This also reproduces with Docker runtime. Workaround with bind mount helps, however automatically unmounts it so don't get confused. is on root that is mounted like this: , and docker is configured to use .
Reproduced with k8s 1.19.3 on Gentoo. The rootfs is btrfs, is on rootfs, docker is configured with storage drigver . automatically mount and unmount it on . Also confirmed bind mount workaround helps.
/assign
I think almost anyone using BTRFS as the root filesystem has encountered this issue. If is merged and applied to the kubernetes, this can be fixed. Please take a look at this ?
The has been merged to google/cadvisor. I think if Kubernetes v1.21 includes the fix by using google/cadvisor v0.39, the issue will be resolved.
Just a note that I too have stumbled on this issue after reconfiguring CRI-O storage to use btrfs driver pointing with graphroot directly to a BTRFS subvolume (the root filesystem is not BTRFS). After reading this issue I tried mounting the BTRFS subvolume explicitly on some mountpoint and pointed with graphroot to this explicit subvolume mountpoint which solved the problem. The problem seems to occur only when pointing to a subvolume within some parent subvolume as this is not a real mountpoint.
Ok, so I just want to make sure I understand. Currently when i try to run kubeadm to create a new cluster while using btrfs, it gives me With Kubernetes v1.21, that will no longer be the case?
In my experience (with CRI-O 1.19 and k8s 1.19), the btrfs driver (in ) is supported, but the combination does not work correctly when graphroot is pointed to a btrfs subvolume within some parent subvolume (I don't have experience with graphroot poited to some subdirectory of a default subvolume and/or when default subvolume is also a root filesystem, just when pointed directly to a subvolume within some parent subvolume which is not a root filesystem). For example, when btrfs filesystem is mounted like the following (): when I then create a subvolume: then use it as a graphroot (): But when I explicitly mount the subvolume to some mountpoint (): then use the explicitly mounted path as the graphroot (): it works correctly without any problems. I guess some logic (either in CRI-O/containers or in kubelet) tries to find the "mountpoint" of the filesystem used for graphroot or something related and can't do that correctly when graphroot is located in a btrfs subvolume that is not explicitly mounted somewhere.
Ah, I see what you meant by saying that is not supported in . I haven't checked that since I created k8s clusted with default (overlay) driver and later migrated to . It is possible that disallows using because of problems like the one described above. But in my experience, when a subvolume is mounted explicitly, the btrfs works as expected - at least I didn't have problems with it.
Can the fix be backported to 1.19? only supports updating one version at a time: () So this leaves people on btrfs with with no upgrade path to which is not really nice.
We've discussed this issue at Regarding Maybe, there are quite a lot of people using btrfs than I expected. What do you think about this? Thanks!
I am running up against this issue as well. I am stuck with using k8s on 1.18
It sounds like for full fix we'll need to cherrypick to be included in k8s 1.19 and 1.20. To do that the following is needed: to cAdvisor v0.37 and v0.38 branches k8s 1.19 uses cAdvisor v0.37.3 - k8s 1.20 uses cAdvisor v0.38.6 - new cAdvisor patch release for v0.37 and v0.38 against k8s on 1.19 and 1.20 bumping to newly released cAdvisor (v0.37 / v0.38) with the fix I'm happy to cut the new cAdvisor release from my side if you want to proceed and think this is critical enough bug that is worth cherrypicking.
New cAdvisor patch versions have been cut with the BTRFS fix: v0.37.4 - v0.38.7 - /cc

Discussion:
github.com/gogo/protobuf v1.3.1 github.com/golang/groupcache v0.0.0-20200121045136-8c9f03a8e57e github.com/golang/mock v1.4.4 <del> github.com/google/cadvisor v0.38.6 </del> <ins> github.com/google/cadvisor v0.38.7 </ins> github.com/google/go-cmp v0.5.2 github.com/google/gofuzz v1.1.0 github.com/google/uuid v1.1.2

github.com/golangplus/fmt => github.com/golangplus/fmt v0.0.0-20150411045040-2a5d6d7d2995 github.com/golangplus/testing => github.com/golangplus/testing v0.0.0-20180327235837-af21d9c3145e github.com/google/btree => github.com/google/btree v1.0.0 <del> github.com/google/cadvisor => github.com/google/cadvisor v0.38.6 </del> <ins> github.com/google/cadvisor => github.com/google/cadvisor v0.38.7 </ins> github.com/google/go-cmp => github.com/google/go-cmp v0.5.2 github.com/google/gofuzz => github.com/google/gofuzz v1.1.0 github.com/google/martian => github.com/google/martian v2.1.0+incompatible

github.com/golangplus/testing v0.0.0-20180327235837-af21d9c3145e/go.mod h1:0AA//k/eakGydO4jKRoRL2j92ZKSzTgj9tclaCrvXHk= github.com/google/btree v1.0.0 h1:0udJVsspx3VBr5FwtLhQQtuAsVc79tTq0ocGIPAU6qo= github.com/google/btree v1.0.0/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ= <del> github.com/google/cadvisor v0.38.6 h1:5vu8NaOqsBKF6wOxBLeDPD7hcmxfg/4I5NZGYXK7gIo= github.com/google/cadvisor v0.38.6/go.mod h1:1OFB9sOOMkBdUBGCO/1SArawTnDscgMzTodacVDe8mA= </del> <ins> github.com/google/cadvisor v0.38.7 h1:ZWyUz+23k1PRmEA+yrnDGtEC6IuU4Vc6439x2NQLHnA= github.com/google/cadvisor v0.38.7/go.mod h1:1OFB9sOOMkBdUBGCO/1SArawTnDscgMzTodacVDe8mA= </ins> github.com/google/go-cmp v0.5.2 h1:X2ev0eStA3AbceY54o37/0PQ/UWqKEiiO2dKL5OPaFM= github.com/google/go-cmp v0.5.2/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= github.com/google/gofuzz v1.1.0 h1:Hsa8mG0dQ46ij8Sl2AYJDUv1oA9/d6Vk+3LG99Oe02g=

return &DeviceInfo{deviceName, p.major, p.minor}, nil } <del> func (i *RealFsInfo) GetDirFsDevice(dir string) (*DeviceInfo, error) { buf := new(syscall.Stat_t) err := syscall.Stat(dir, buf) if err != nil { return nil, fmt.Errorf("stat failed on %s with error: %s", dir, err) } // The type Dev in Stat_t is 32bit on mips. major := major(uint64(buf.Dev)) // nolint: unconvert minor := minor(uint64(buf.Dev)) // nolint: unconvert for device, partition := range i.partitions { if partition.major == major && partition.minor == minor { return &DeviceInfo{device, major, minor}, nil } } </del> <ins> func (i *RealFsInfo) mountInfoFromDir(dir string) (*mount.MountInfo, bool) { </ins> mount, found := i.mounts[dir] // try the parent dir if not found until we reach the root dir // this is an issue on btrfs systems where the directory is not

pathdir, _ := filepath.Split(dir) // break when we reach root if pathdir == "/" { <ins> mount, found = i.mounts["/"] </ins> break } // trim "/" from the new parent path otherwise the next possible

dir = strings.TrimSuffix(pathdir, "/") mount, found = i.mounts[dir] } <ins> return &mount, found } func (i *RealFsInfo) GetDirFsDevice(dir string) (*DeviceInfo, error) { buf := new(syscall.Stat_t) err := syscall.Stat(dir, buf) if err != nil { return nil, fmt.Errorf("stat failed on %s with error: %s", dir, err) } // The type Dev in Stat_t is 32bit on mips. major := major(uint64(buf.Dev)) // nolint: unconvert minor := minor(uint64(buf.Dev)) // nolint: unconvert for device, partition := range i.partitions { if partition.major == major && partition.minor == minor { return &DeviceInfo{device, major, minor}, nil } } mount, found := i.mountInfoFromDir(dir) </ins> if found && mount.FsType == "btrfs" && mount.Major == 0 && strings.HasPrefix(mount.Source, "/dev/") { <del> major, minor, err := getBtrfsMajorMinorIds(&mount) </del> <ins> major, minor, err := getBtrfsMajorMinorIds(mount) </ins> if err != nil { klog.Warningf("%s", err) } else {

# github.com/google/btree v1.0.0 => github.com/google/btree v1.0.0 github.com/google/btree # github.com/google/btree => github.com/google/btree v1.0.0 <del> # github.com/google/cadvisor v0.38.6 => github.com/google/cadvisor v0.38.6 </del> <ins> # github.com/google/cadvisor v0.38.7 => github.com/google/cadvisor v0.38.7 </ins> ## explicit <del> # github.com/google/cadvisor => github.com/google/cadvisor v0.38.6 </del> <ins> # github.com/google/cadvisor => github.com/google/cadvisor v0.38.7 </ins> github.com/google/cadvisor/accelerators github.com/google/cadvisor/cache/memory github.com/google/cadvisor/client/v2