Change:
Kubernetes should resize filesystem on a volume when it's created as a clone of a smaller volume / smaller snapshot is restored into it. Example: takes a snapshot of 1 GiB volume with ext3 filesystem on it. restores the snapshot into 2 GiB volume. 2.1 CSI driver restores the snapshot into a bigger volume, but ext3 filesystem still uses only 1 GiB of space. driver should make sure that the filesystem on the volume has been resized before the volume is mounted into a pod. There are few steps needed: [x] Clarify behavior of CSI call, [ ] Update CSI drivers to the new behavior. [x] Move filesystem resize code from kubernetes/kuberenetes to kubernetes/mount-utils, so all CSI drivers can re-use the same code [x] Enhance the code above to compare the volume size from block device perspective and from filesystem perspective. See for proof of concept. [ ] File issues in all community CSI drivers that we care about & fix them. [ ] Update csi-tests to (optionally?) check for volume size when cloning / restoring snapshot to a bigger size.
/assign /sig storage /priority important-soon
True, considering the CSI spec allows restore to a bigger size volume, we should have the support from CO.
This is hard, Kubernets PV controller overwrites on binding. Therefore the external-provisioner cannot create PVC as "marked for resize on node", i.e. with = = the real size of the new volume) and with = size of the original volume. We either need to change the binding mechanism not to overwrite or add a flag (annotation?) that marks the volume for resize on node, even when .
Talked to The external-provisioner could mark for resize on node: Set to the original volume size. Add condition to the PVC. As result, the extenal-provisioner needs to modify , which needs new RBAC and is a breaking change. The PV controller could respect , if set. We discussed and rejected adding a new flag / annotation to PV by the external-provisioner for kubelet to call NodeExpandVolume on the first mount. Such a flag would need to be cleared by kubelet and kubelet can't modify PVs.
From latest discussion last week, I believe the conclusion was to have external-provisioner set to the smaller size. Then we just let the rest of the normal resize flow continue. No need to have provisioner set the conditions.
we agreed to set conditions too because we want to be as close as possible to external-resizer. If there is some tooling that depends on condition for resizing pending on node, we don't want to break any such tooling. currently kubelet does not depend on condition for node expansion though.
Why can't we let resize controller set the condition when it sees pvc size != pv size?
hmm good point. I think that should work but we may have to verify that flow.
I think we can do that, but only with a change to the CSI spec wording, because the current implication is that should not be necessary.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
can you summarize the current conclusion/state of this? /remove-lifecycle rotten
Updated the issue with TODO list.
should this issue stay open to keep track of remaining issues?
/reopen
Reopened this issue. <detailsIn response to : /reopen </details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
/triage accepted
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle rotten
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle stale /reopen
Reopened this issue. <detailsIn response to : /remove-lifecycle stale /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The issue has been marked as an important bug and triaged. Such issues are automatically marked as frozen when hitting the rotten state to avoid missing important bugs. Please send feedback to sig-contributor-experience at . /lifecycle frozen
This issue is labeled with but has not been updated in over 90 days, and should be re-triaged. Important-soon issues must be staffed and worked on either currently, or very soon, ideally in time for the next release. You can: Confirm that this issue is still relevant with (org members only) Deprioritize it with or Close this issue with For more details on the triage process, see /remove-triage accepted
/triage accepted /priority long-term what is the latest status on this?
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /triage accepted /priority long-term what is the latest status on this? Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/priority important-long-term
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /priority important-long-term Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hello I've checked what is the current status of the drivers that were involved in this effort and I think we might be able to close this as all upstream work is done. Please take a look at the table below and let me know if anything is missing.
/close
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

Discussion:
import ( "fmt" <ins> "strconv" "strings" </ins> "k8s.io/klog/v2" utilexec "k8s.io/utils/exec"

resizeError := fmt.Errorf("resize of device %s failed: %v. btrfs output: %s", deviceMountPath, err, string(output)) return false, resizeError } <ins> func (resizefs *ResizeFs) NeedResize(devicePath string, deviceMountPath string) (bool, error) { deviceSize, err := resizefs.getDeviceSize(devicePath) if err != nil { return false, err } var fsSize, blockSize uint64 format, err := getDiskFormat(resizefs.exec, devicePath) if err != nil { formatErr := fmt.Errorf("ResizeFS.Resize - error checking format for device %s: %v", devicePath, err) return false, formatErr } // If disk has no format, there is no need to resize the disk because mkfs.* // by default will use whole disk anyways. if format == "" { return false, nil } klog.V(3).Infof("ResizeFs.needResize - checking mounted volume %s", devicePath) switch format { case "ext3", "ext4": blockSize, fsSize, err = resizefs.getExtSize(devicePath) klog.V(5).Infof("Ext size: filesystem size=%d, block size=%d", fsSize, blockSize) case "xfs": blockSize, fsSize, err = resizefs.getXFSSize(deviceMountPath) klog.V(5).Infof("Xfs size: filesystem size=%d, block size=%d, err=%v", fsSize, blockSize, err) default: klog.Errorf("Not able to parse given filesystem info. fsType: %s, will not resize", format) return false, fmt.Errorf("Could not parse fs info on given filesystem format: %s. Supported fs types are: xfs, ext3, ext4", format) } if err != nil { return false, err } // Tolerate one block difference, just in case of rounding errors somewhere. klog.V(5).Infof("Volume %s: device size=%d, filesystem size=%d, block size=%d", devicePath, deviceSize, fsSize, blockSize) if deviceSize <= fsSize+blockSize { return false, nil } return true, nil } func (resizefs *ResizeFs) getDeviceSize(devicePath string) (uint64, error) { output, err := resizefs.exec.Command("blockdev", "--getsize64", devicePath).CombinedOutput() outStr := strings.TrimSpace(string(output)) if err != nil { return 0, fmt.Errorf("failed to read size of device %s: %s: %s", devicePath, err, outStr) } size, err := strconv.ParseUint(outStr, 10, 64) if err != nil { return 0, fmt.Errorf("failed to parse size of device %s %s: %s", devicePath, outStr, err) } return size, nil } func (resizefs *ResizeFs) getExtSize(devicePath string) (uint64, uint64, error) { output, err := resizefs.exec.Command("dumpe2fs", "-h", devicePath).CombinedOutput() if err != nil { return 0, 0, fmt.Errorf("failed to read size of filesystem on %s: %s: %s", devicePath, err, string(output)) } blockSize, blockCount, _ := resizefs.parseFsInfoOutput(string(output), ":", "block size", "block count") if blockSize == 0 { return 0, 0, fmt.Errorf("could not find block size of device %s", devicePath) } if blockCount == 0 { return 0, 0, fmt.Errorf("could not find block count of device %s", devicePath) } return blockSize, blockSize * blockCount, nil } func (resizefs *ResizeFs) getXFSSize(devicePath string) (uint64, uint64, error) { output, err := resizefs.exec.Command("xfs_io", "-c", "statfs", devicePath).CombinedOutput() if err != nil { return 0, 0, fmt.Errorf("failed to read size of filesystem on %s: %s: %s", devicePath, err, string(output)) } blockSize, blockCount, _ := resizefs.parseFsInfoOutput(string(output), "=", "geom.bsize", "geom.datablocks") if blockSize == 0 { return 0, 0, fmt.Errorf("could not find block size of device %s", devicePath) } if blockCount == 0 { return 0, 0, fmt.Errorf("could not find block count of device %s", devicePath) } return blockSize, blockSize * blockCount, nil } func (resizefs *ResizeFs) parseFsInfoOutput(cmdOutput string, spliter string, blockSizeKey string, blockCountKey string) (uint64, uint64, error) { lines := strings.Split(cmdOutput, "n") var blockSize, blockCount uint64 var err error for _, line := range lines { tokens := strings.Split(line, spliter) if len(tokens) != 2 { continue } key, value := strings.ToLower(strings.TrimSpace(tokens[0])), strings.ToLower(strings.TrimSpace(tokens[1])) if key == blockSizeKey { blockSize, err = strconv.ParseUint(value, 10, 64) if err != nil { return 0, 0, fmt.Errorf("failed to parse block size %s: %s", value, err) } } if key == blockCountKey { blockCount, err = strconv.ParseUint(value, 10, 64) if err != nil { return 0, 0, fmt.Errorf("failed to parse block count %s: %s", value, err) } } } return blockSize, blockCount, err } </ins>

<ins> // +build linux /* Copyright 2021 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package mount import ( "k8s.io/utils/exec" fakeexec "k8s.io/utils/exec/testing" "testing" ) func TestGetFileSystemSize(t *testing.T) { cmdOutputSuccessXfs := ` statfs.f_bsize = 4096 statfs.f_blocks = 1832448 statfs.f_bavail = 1822366 statfs.f_files = 3670016 statfs.f_ffree = 3670012 statfs.f_flags = 0x1020 geom.bsize = 4096 geom.agcount = 4 geom.agblocks = 458752 geom.datablocks = 1835008 geom.rtblocks = 0 geom.rtextents = 0 geom.rtextsize = 1 geom.sunit = 0 geom.swidth = 0 counts.freedata = 1822372 counts.freertx = 0 counts.freeino = 61 counts.allocino = 64 ` cmdOutputNoDataXfs := ` statfs.f_bsize = 4096 statfs.f_blocks = 1832448 statfs.f_bavail = 1822366 statfs.f_files = 3670016 statfs.f_ffree = 3670012 statfs.f_flags = 0x1020 geom.agcount = 4 geom.agblocks = 458752 geom.rtblocks = 0 geom.rtextents = 0 geom.rtextsize = 1 geom.sunit = 0 geom.swidth = 0 counts.freedata = 1822372 counts.freertx = 0 counts.freeino = 61 counts.allocino = 64 ` cmdOutputSuccessExt4 := ` Filesystem volume name:   cloudimg-rootfs Last mounted on:          / Filesystem UUID:          testUUID Filesystem magic number:  0xEF53 Filesystem revision #:    1 (dynamic) Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery extent 64bit Default mount options:    user_xattr acl Filesystem state:         clean Errors behavior:          Continue Filesystem OS type:       Linux Inode count:              3840000 Block count:              5242880 Reserved block count:     0 Free blocks:              5514413 Free inodes:              3677492 First block:              0 Block size:               4096 Fragment size:            4096 Group descriptor size:    64 Reserved GDT blocks:      252 Blocks per group:         32768 Fragments per group:      32768 Inodes per group:         16000 Inode blocks per group:   1000 Flex block group size:    16 Mount count:              2 Maximum mount count:      -1 Check interval:           0 (<none>) Lifetime writes:          180 GB Reserved blocks uid:      0 (user root) Reserved blocks gid:      0 (group root) First inode:              11 Inode size:	              256 Required extra isize:     32 Desired extra isize:      32 Journal inode:            8 Default directory hash:   half_md4 Directory Hash Seed:      Test Hashing Journal backup:           inode blocks Checksum type:            crc32c Checksum:                 0x57705f62 Journal features:         journal_incompat_revoke journal_64bit journal_checksum_v3 Journal size:             64M Journal length:           16384 Journal sequence:         0x00037109 Journal start:            1 Journal checksum type:    crc32c Journal checksum:         0xb7df3c6e ` cmdOutputNoDataExt4 := `Filesystem volume name:   cloudimg-rootfs Last mounted on:          / Filesystem UUID:          testUUID Filesystem magic number:  0xEF53 Filesystem revision #:    1 (dynamic) Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery extent 64bit Default mount options:    user_xattr acl Filesystem state:         clean Errors behavior:          Continue Filesystem OS type:       Linux Inode count:              3840000 Reserved block count:     0 Free blocks:              5514413 Free inodes:              3677492 First block:              0 Fragment size:            4096 Group descriptor size:    64 Reserved GDT blocks:      252 Blocks per group:         32768 Fragments per group:      32768 Inodes per group:         16000 Inode blocks per group:   1000 Flex block group size:    16 Mount count:              2 Maximum mount count:      -1 Check interval:           0 (<none>) Lifetime writes:          180 GB Reserved blocks uid:      0 (user root) Reserved blocks gid:      0 (group root) First inode:              11 Inode size:	              256 Required extra isize:     32 Desired extra isize:      32 Journal inode:            8 Default directory hash:   half_md4 Directory Hash Seed:      Test Hashing Journal backup:           inode blocks Checksum type:            crc32c Checksum:                 0x57705f62 Journal features:         journal_incompat_revoke journal_64bit journal_checksum_v3 Journal size:             64M Journal length:           16384 Journal sequence:         0x00037109 Journal start:            1 Journal checksum type:    crc32c Journal checksum:         0xb7df3c6e ` testcases := []struct { name        string devicePath  string blocksize   uint64 blockCount  uint64 cmdOutput   string expectError bool fsType      string }{ { name:        "success parse xfs info", devicePath:  "/dev/test1", blocksize:   4096, blockCount:  1835008, cmdOutput:   cmdOutputSuccessXfs, expectError: false, fsType:      "xfs", }, { name:        "block size not present - xfs", devicePath:  "/dev/test1", blocksize:   0, blockCount:  0, cmdOutput:   cmdOutputNoDataXfs, expectError: true, fsType:      "xfs", }, { name:        "success parse ext info", devicePath:  "/dev/test1", blocksize:   4096, blockCount:  5242880, cmdOutput:   cmdOutputSuccessExt4, expectError: false, fsType:      "ext4", }, { name:        "block size not present - ext4", devicePath:  "/dev/test1", blocksize:   0, blockCount:  0, cmdOutput:   cmdOutputNoDataExt4, expectError: true, fsType:      "ext4", }, } for _, test := range testcases { t.Run(test.name, func(t *testing.T) { fcmd := fakeexec.FakeCmd{ CombinedOutputScript: []fakeexec.FakeAction{ func() ([]byte, []byte, error) { return []byte(test.cmdOutput), nil, nil }, }, } fexec := fakeexec.FakeExec{ CommandScript: []fakeexec.FakeCommandAction{ func(cmd string, args ...string) exec.Cmd { return fakeexec.InitFakeCmd(&fcmd, cmd, args...) }, }, } resizefs := ResizeFs{exec: &fexec} var blockSize uint64 var fsSize uint64 var err error switch test.fsType { case "xfs": blockSize, fsSize, err = resizefs.getXFSSize(test.devicePath) case "ext4": blockSize, fsSize, err = resizefs.getExtSize(test.devicePath) } if blockSize != test.blocksize { t.Fatalf("Parse wrong block size value, expect %d, but got %d", test.blocksize, blockSize) } if fsSize != test.blocksize*test.blockCount { t.Fatalf("Parse wrong fs size value, expect %d, but got %d", test.blocksize*test.blockCount, fsSize) } if !test.expectError && err != nil { t.Fatalf("Expect no error but got %v", err) } }) } } func TestNeedResize(t *testing.T) { testcases := []struct { name            string devicePath      string deviceMountPath string deviceSize      string cmdOutputFsType string expectError     bool expectResult    bool }{ { name:            "False - Unsupported fs type", devicePath:      "/dev/test1", deviceMountPath: "/mnt/test1", deviceSize:      "2048", cmdOutputFsType: "TYPE=ntfs", expectError:     true, expectResult:    false, }, } for _, test := range testcases { t.Run(test.name, func(t *testing.T) { fcmd := fakeexec.FakeCmd{ CombinedOutputScript: []fakeexec.FakeAction{ func() ([]byte, []byte, error) { return []byte(test.deviceSize), nil, nil }, func() ([]byte, []byte, error) { return []byte(test.cmdOutputFsType), nil, nil }, }, } fexec := fakeexec.FakeExec{ CommandScript: []fakeexec.FakeCommandAction{ func(cmd string, args ...string) exec.Cmd { return fakeexec.InitFakeCmd(&fcmd, cmd, args...) }, func(cmd string, args ...string) exec.Cmd { return fakeexec.InitFakeCmd(&fcmd, cmd, args...) }, }, } resizefs := ResizeFs{exec: &fexec} needResize, err := resizefs.NeedResize(test.devicePath, test.deviceMountPath) if needResize != test.expectResult { t.Fatalf("Expect result is %v but got %v", test.expectResult, needResize) } if !test.expectError && err != nil { t.Fatalf("Expect no error but got %v", err) } }) } } </ins>