Change:
After upgrading our clusters to Kubernetes 1.19, the provisioning of PersistentVolumes started failing with the following Events: The value matches the actual storage request of 3.2Gi. It doesn't seem special in any way, not too big, not too small, and it always worked until 1.19. We look into the issue with and we believe the PR introduced a change that broke the PersistentVolume provisioning even when it doesn’t overflow int64. Amounts of resource in Kubernetes are represented with the . In the PersistentVolume drivers, there are some functions that manipulate storage request in order to allocate a disk with the right size. Before the mentioned PR, those quantities were retrieved by calling which relies on to return the value as an int64. This worked probably fine in most of the cases, but it could happen that the represented value would overflow an int64, and in this case the result of would be wrong. The function’s description states that it’s the caller’s responsibility to check this: This is likely what the PR wanted to handle. The change replaced all calls with which has a different signature. From where it’s used now, you would think that the second return value signals an overflow: : This is probably where the error we’re seeing is generated. I believe that the signature of was misinterpreted. From my understanding of its description, returning doesn’t mean we’re overflowing . It only means that internally, the quantity is represented as a decimal amount: Internally, depending on their values, quantities are stored/represented in : for DecimalExponent (e.g. ) or DecimalSI (e.g. ) for BinarySI that don’t have a decimal part but a unit (simplified) (e.g. ) for all the rest With the PR, Kubernetes now fails to provision volumes that have a storage request represented as . Many of the storage requests are represented this way and still perfectly valid in our opinion. I would have expected the provisioning to continue to work as those requests seemed valid and were always accepted by our underlying cloud-provider in earlier version of Kubernetes. Create a StatefulSet with a PersistentVolumeClaim that uses vSphere storage: Kubernetes version (use ): 1.19.7 Cloud provider or hardware configuration: vSphere, but I think it impacts other cloud-provider that were changed in the PR: /sig storage /kind bug /cc Possibly related to
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I think the only reason we do not use the quantity library for scaling and why we tried to roll our own rounding code is because when scaling stuff in AsScaledInt64(3) or ScaledValue(3), the library assumes decimal SI (M, 1000), not binary SI units (Mi, 1024): You are right, our self-rolled rounding code doesn't handle values with millibyte or higher precision like 3.2Gi. Our code uses AsInt64 and then performs its own scaling. If AsInt64 returns false, the int64 returned is not in bytes (it must be in millibytes or higher precision), so our rounding code won't work, so it returns that " is too great, overflows int64". In 1.18, we used Value() to round to bytes but we didn't check it against ScaleValue as the signature suggested. So the PR in question fixed cases where Value() overflowed but inadvertently broke your case. The correct way to round, which the resource library's rounding does, is to drop the useless precision and round up. , but as I said it doesn't seem to handle binary SI quantities correctly. For decimal SI quantities, we should rely on the resource library's AsScaledInt64 and be done with it. (We could also use Value() and ScaledValue as the signature suggests but calling AsScaledInt64 directly seems simpler). Like this: For binary SI quantities, we need to do something fancier involving CanonicalizeBytes I think, I'm not sure yet.
I guess for binary SI we can go back to using Value but actually check the overflow this time.
Hi Thanks for the quick feedback on the problem's description already. Indeed, the Scaled() functions don't handle BinarySI and that's why I quickly discarded the idea of using them. Since this is about storage, I suppose most people are using this format. I just finished wrapping up a PR that I hope address both the issue we are facing, but also what was meant to be fixed in the PR your reviewed back then. It doesn't make any difference between the 3 formats however.
Yeah I commented within seconds of your PR. I glanced at it , will review more soon, but it makes sense to me to just use VAlue and check ofr overflow for both decimal SI and binary SI rather than using Scaled() for one and not the other :+1:
I think it means both. i.e - when incoming value is decimal or if value is so big that type can't store it as it switches to decimal. So while my PR fixed later case, it broke the former. :/

Discussion:
} vol := vols[0] <del> tBytes, ok := newSize.AsInt64() if !ok { return fmt.Errorf("quantity %v is too great, overflows int64", newSize) </del> <ins> tBytes, err := volumehelpers.RoundUpToB(newSize) if err != nil { return err </ins> } newSizeInBytes := uint64(tBytes) if vol.Spec.Size >= newSizeInBytes {

import ( "fmt" <ins> "math" "math/bits" </ins> "k8s.io/apimachinery/pkg/api/resource" ) <ins> /* The Cloud Provider's volume plugins provision disks for corresponding PersistentVolumeClaims. Cloud Providers use different allocation unit for their disk sizes. AWS allows you to specify the size as an integer amount of GiB, while Portworx expects bytes for example. On AWS, if you want a volume of 1500MiB, the actual call to the AWS API should therefore be for a 2GiB disk. This file contains functions that help rounding a storage request based on a Cloud Provider's allocation unit. */ </ins> const ( // GB - GigaByte size GB = 1000 * 1000 * 1000

// RoundUpToGiB rounds up given quantity upto chunks of GiB func RoundUpToGiB(size resource.Quantity) (int64, error) { <del> requestBytes, ok := size.AsInt64() if !ok { return 0, fmt.Errorf("quantity %s is too great, overflows int64", size.String()) } return roundUpSize(requestBytes, GiB), nil </del> <ins> return roundUpSizeInt64(size, GiB) </ins> } // RoundUpToMB rounds up given quantity to chunks of MB func RoundUpToMB(size resource.Quantity) (int64, error) { <del> requestBytes, ok := size.AsInt64() if !ok { return 0, fmt.Errorf("quantity %s is too great, overflows int64", size.String()) } return roundUpSize(requestBytes, MB), nil </del> <ins> return roundUpSizeInt64(size, MB) </ins> } // RoundUpToMiB rounds up given quantity upto chunks of MiB func RoundUpToMiB(size resource.Quantity) (int64, error) { <del> requestBytes, ok := size.AsInt64() if !ok { return 0, fmt.Errorf("quantity %s is too great, overflows int64", size.String()) } return roundUpSize(requestBytes, MiB), nil </del> <ins> return roundUpSizeInt64(size, MiB) </ins> } // RoundUpToKB rounds up given quantity to chunks of KB func RoundUpToKB(size resource.Quantity) (int64, error) { <del> requestBytes, ok := size.AsInt64() if !ok { return 0, fmt.Errorf("quantity %s is too great, overflows int64", size.String()) } return roundUpSize(requestBytes, KB), nil </del> <ins> return roundUpSizeInt64(size, KB) </ins> } <del> // RoundUpToKiB rounds up given quantity upto chunks of KiB </del> <ins> // RoundUpToKiB rounds up given quantity to chunks of KiB </ins> func RoundUpToKiB(size resource.Quantity) (int64, error) { <del> requestBytes, ok := size.AsInt64() if !ok { return 0, fmt.Errorf("quantity %s is too great, overflows int64", size.String()) } return roundUpSize(requestBytes, KiB), nil </del> <ins> return roundUpSizeInt64(size, KiB) } // RoundUpToB rounds up given quantity to chunks of bytes func RoundUpToB(size resource.Quantity) (int64, error) { return roundUpSizeInt64(size, 1) </ins> } // RoundUpToGiBInt rounds up given quantity upto chunks of GiB. It returns an // int instead of an int64 and an error if there's overflow func RoundUpToGiBInt(size resource.Quantity) (int, error) { <del> requestBytes, ok := size.AsInt64() if !ok { return 0, fmt.Errorf("quantity %s is too great, overflows int64", size.String()) } return roundUpSizeInt(requestBytes, GiB) </del> <ins> return roundUpSizeInt(size, GiB) </ins> } // RoundUpToMBInt rounds up given quantity to chunks of MB. It returns an // int instead of an int64 and an error if there's overflow func RoundUpToMBInt(size resource.Quantity) (int, error) { <del> requestBytes, ok := size.AsInt64() if !ok { return 0, fmt.Errorf("quantity %s is too great, overflows int64", size.String()) } return roundUpSizeInt(requestBytes, MB) </del> <ins> return roundUpSizeInt(size, MB) </ins> } // RoundUpToMiBInt rounds up given quantity upto chunks of MiB. It returns an // int instead of an int64 and an error if there's overflow func RoundUpToMiBInt(size resource.Quantity) (int, error) { <del> requestBytes, ok := size.AsInt64() if !ok { return 0, fmt.Errorf("quantity %s is too great, overflows int64", size.String()) } return roundUpSizeInt(requestBytes, MiB) </del> <ins> return roundUpSizeInt(size, MiB) </ins> } // RoundUpToKBInt rounds up given quantity to chunks of KB. It returns an // int instead of an int64 and an error if there's overflow func RoundUpToKBInt(size resource.Quantity) (int, error) { <del> requestBytes, ok := size.AsInt64() if !ok { return 0, fmt.Errorf("quantity %s is too great, overflows int64", size.String()) } return roundUpSizeInt(requestBytes, KB) </del> <ins> return roundUpSizeInt(size, KB) </ins> } // RoundUpToKiBInt rounds up given quantity upto chunks of KiB. It returns an // int instead of an int64 and an error if there's overflow func RoundUpToKiBInt(size resource.Quantity) (int, error) { <del> requestBytes := size.Value() return roundUpSizeInt(requestBytes, KiB) </del> <ins> return roundUpSizeInt(size, KiB) </ins> } // RoundUpToGiBInt32 rounds up given quantity up to chunks of GiB. It returns an // int32 instead of an int64 and an error if there's overflow func RoundUpToGiBInt32(size resource.Quantity) (int32, error) { <del> requestBytes, ok := size.AsInt64() if !ok { return 0, fmt.Errorf("quantity %s is too great, overflows int64", size.String()) } return roundUpSizeInt32(requestBytes, GiB) </del> <ins> return roundUpSizeInt32(size, GiB) </ins> } // roundUpSizeInt calculates how many allocation units are needed to accommodate <del> // a volume of given size. It returns an int instead of an int64 and an error if // there's overflow func roundUpSizeInt(volumeSizeBytes int64, allocationUnitBytes int64) (int, error) { roundedUp := roundUpSize(volumeSizeBytes, allocationUnitBytes) roundedUpInt := int(roundedUp) if int64(roundedUpInt) != roundedUp { return 0, fmt.Errorf("capacity %v is too great, casting results in integer overflow", roundedUp) </del> <ins> // a volume of a given size. It returns an int and an error if there's overflow func roundUpSizeInt(size resource.Quantity, allocationUnitBytes int64) (int, error) { if bits.UintSize == 32 { res, err := roundUpSizeInt32(size, allocationUnitBytes) return int(res), err </ins> } <del> return roundedUpInt, nil </del> <ins> res, err := roundUpSizeInt64(size, allocationUnitBytes) return int(res), err </ins> } // roundUpSizeInt32 calculates how many allocation units are needed to accommodate <del> // a volume of given size. It returns an int32 instead of an int64 and an error if // there's overflow func roundUpSizeInt32(volumeSizeBytes int64, allocationUnitBytes int64) (int32, error) { roundedUp := roundUpSize(volumeSizeBytes, allocationUnitBytes) roundedUpInt32 := int32(roundedUp) if int64(roundedUpInt32) != roundedUp { return 0, fmt.Errorf("quantity %v is too great, overflows int32", roundedUp) </del> <ins> // a volume of a given size. It returns an int32 and an error if there's overflow func roundUpSizeInt32(size resource.Quantity, allocationUnitBytes int64) (int32, error) { roundedUpInt32, err := roundUpSizeInt64(size, allocationUnitBytes) if err != nil { return 0, err } if roundedUpInt32 > math.MaxInt32 { return 0, fmt.Errorf("quantity %s is too great, overflows int32", size.String()) </ins> } <del> return roundedUpInt32, nil </del> <ins> return int32(roundedUpInt32), nil </ins> } <del> // roundUpSize calculates how many allocation units are needed to accommodate // a volume of given size. E.g. when user wants 1500MiB volume, while AWS EBS // allocates volumes in gibibyte-sized chunks, // RoundUpSize(1500 * 1024*1024, 1024*1024*1024) returns '2' // (2 GiB is the smallest allocatable volume that can hold 1500MiB) func roundUpSize(volumeSizeBytes int64, allocationUnitBytes int64) int64 { </del> <ins> // roundUpSizeInt64 calculates how many allocation units are needed to accommodate // a volume of a given size. It returns an int64 and an error if there's overflow func roundUpSizeInt64(size resource.Quantity, allocationUnitBytes int64) (int64, error) { // Use CmpInt64() to find out if the value of "size" would overflow an // int64 and therefore have Value() return a wrong result. Then, retrieve // the value as int64 and perform the rounding. // It's not convenient to use AsScale() and related functions as they don't // support BinarySI format, nor can we use AsInt64() directly since it's // only implemented for int64 scaled numbers (int64Amount). // CmpInt64() actually returns 0 when comparing an amount bigger than MaxInt64. if size.CmpInt64(math.MaxInt64) >= 0 { return 0, fmt.Errorf("quantity %s is too great, overflows int64", size.String()) } volumeSizeBytes := size.Value() </ins> roundedUp := volumeSizeBytes / allocationUnitBytes if volumeSizeBytes%allocationUnitBytes > 0 { roundedUp++ } <del> return roundedUp </del> <ins> return roundedUp, nil </ins> }

roundedVal: int64(1000), }, { <del> name:        "round overflowed quantity to int64", resource:    resource.MustParse("73786976299133170k"), </del> <ins> name:       "round decimal to GiB", resource:   resource.MustParse("1.2Gi"), roundedVal: int64(2), }, { name:       "round big value to GiB", resource:   resource.MustParse("8191Pi"), roundedVal: int64(8588886016), }, { name:        "round quantity to GiB that would lead to an int64 overflow", resource:    resource.MustParse("8192Pi"), </ins> roundedVal:  int64(0), expectError: true, },

roundedVal: int64(1073742), }, { <del> name:        "round overflowed quantity to int64", resource:    resource.MustParse("73786976299133170k"), </del> <ins> name:       "round decimal to MB", resource:   resource.MustParse("1.2Gi"), roundedVal: int64(1289), }, { name:       "round big value to MB", resource:   resource.MustParse("8191Pi"), roundedVal: int64(9222246136948), }, { name:        "round quantity to MB that would lead to an int64 overflow", resource:    resource.MustParse("8192Pi"), </ins> roundedVal:  int64(0), expectError: true, },

roundedVal: int64(1024000), }, { <del> name:        "round overflowed quantity to int64", resource:    resource.MustParse("73786976299133170k"), </del> <ins> name:       "round decimal to MiB", resource:   resource.MustParse("1.2Gi"), roundedVal: int64(1229), }, { name:       "round big value to MiB", resource:   resource.MustParse("8191Pi"), roundedVal: int64(8795019280384), }, { name:        "round quantity to MiB that would lead to an int64 overflow", resource:    resource.MustParse("8192Pi"), </ins> roundedVal:  int64(0), expectError: true, },

roundedVal: int64(1073741824), }, { <del> name:        "round overflowed quantity to int64", resource:    resource.MustParse("73786976299133170k"), </del> <ins> name:       "round decimal to KB", resource:   resource.MustParse("1.2Gi"), roundedVal: int64(1288491), }, { name:       "round big value to KB", resource:   resource.MustParse("8191Pi"), roundedVal: int64(9222246136947934), }, { name:        "round quantity to KB that would lead to an int64 overflow", resource:    resource.MustParse("8192Pi"), </ins> roundedVal:  int64(0), expectError: true, },

roundedVal: int64(1048576000), }, { <del> name:        "round overflowed quantity to int64", resource:    resource.MustParse("73786976299133170k"), </del> <ins> name:       "round decimal to KiB", resource:   resource.MustParse("1.2Gi"), roundedVal: int64(1258292), }, { name:       "round big value to KiB", resource:   resource.MustParse("8191Pi"), roundedVal: int64(9006099743113216), }, { name:        "round quantity to KiB that would lead to an int64 overflow", resource:    resource.MustParse("8192Pi"), </ins> roundedVal:  int64(0), expectError: true, },

roundedVal: int32(1000), }, { <del> name:        "round overflowed quantity to int32", resource:    resource.MustParse("73786976299133170k"), </del> <ins> name:       "round decimal to GiB", resource:   resource.MustParse("1.2Gi"), roundedVal: int32(2), }, { name:       "round big value to GiB", resource:   resource.MustParse("2047Pi"), roundedVal: int32(2146435072), }, { name:        "round quantity to GiB that would lead to an int32 overflow", resource:    resource.MustParse("2048Pi"), </ins> roundedVal:  int32(0), expectError: true, },

}) } } <ins> func Test_RoundUpToB(t *testing.T) { testcases := []struct { name        string resource    resource.Quantity roundedVal  int64 expectError bool }{ { name:       "round m to B", resource:   resource.MustParse("987m"), roundedVal: int64(1), }, { name:       "round decimal to B", resource:   resource.MustParse("1.2Gi"), roundedVal: int64(1288490189), }, { name:       "round big value to B", resource:   resource.MustParse("8191Pi"), roundedVal: int64(9222246136947933184), }, { name:        "round quantity to B that would lead to an int64 overflow", resource:    resource.MustParse("8192Pi"), roundedVal:  int64(0), expectError: true, }, } for _, test := range testcases { t.Run(test.name, func(t *testing.T) { val, err := RoundUpToB(test.resource) if !test.expectError && err != nil { t.Errorf("expected no error got: %v", err) } if test.expectError && err == nil { t.Errorf("expected error but got nothing") } if val != test.roundedVal { t.Logf("actual rounded value: %d", val) t.Logf("expected rounded value: %d", test.roundedVal) t.Error("unexpected rounded value") } }) } } </ins>