Code change:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: kubectl create -f ceph- and result is ok now i want to resize pv from 1G to 2G, edit kubectl describe pvc pvc-expand-test What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: ceph storageclass yaml: ceph secret: Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.5", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-11-26T14:41:50Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.5", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-11-26T14:31:35Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration:OS (e.g. from /etc/os-release): CentOS Linux release 7.4.1708 (Core) Kernel (e.g. ): Linux m01 3.10.0-693.el7.x8664 SMP Tue Aug 22 21:09:27 UTC 2017 x8664 x8664 x8664 GNU/Linux Install tools: kubeadm Others: ceph version 13.2.2 () mimic (stable) <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
What's the version of your ceph cluster? k8s uses the command to get rbd image's size, and it expects the size unit in the output is . However, in your environment the unit is :
ceph version 13.2.2 () mimic (stable)
It seems that the output format is different between different ceph versions. I will try to fix it.
/assign
your PR has been merged to master branch, and when will Kubernetes v1.11 fix this bug?
Sorry for the late, I've send cherry-pick PRs:
It seems this change is not working in some cases: I think this is due to two concurrent conditions: 1- ceph client still complain on stderr about not being able to load config file even if there is no need as a key is supplied and the result is (hence the output on stderr of ). This happens not only in hyperkube. 2- it seems we capture both stdout and stderr from the execution of the command (please note the output in the error from ) and then there is then a problem trying to unmarshall the JSON (to be clear, the unuseful warning is on stderr, the json output on stdout). I am able to have a workaround creating empty files for and so that the ceph client does not output anymore but I feel it would be much better to parse only stdout. A reference on the source code for Thanks edit:typo
could you tell where did you put the empty and ceph.keyring files? On the node running kubelet?
Inside the pod of the as the rbd resize is executed by this component.
I had the same problem, I ceph. conf and ceph. Keyring and solved it, but I think k8s got the default user of CEPH before going back to ceph. conf and ceph. Keyring. The following code validates my conjecture: //num 672 Ultimately, I want k8s to get those two files through configurable CEPH parameters. What do you think?
as far as I know you do not need to configure those two files if you use a key. That is still an hypothesis as in the version of Ceph we are using this switch seems to be undocumented: Anyway the warning can be safely ignored and I do not see why we should parse when there is no error
I two files with empty content before, no error; later I a soft connection called and ceph. Keyring (pointing to the ceph configuration file and key ring generated by my cluster), but ceph did not report an error. . I think ceph only needs two files to exist and doesn't care about its contents. Fortunately, the current measures can solve the problem.
this matches my experience (see above, I created the empty files too and it gives no error) but another possibility is just not parse which I find "cleaner" as there is no error in this case even from the error code, and even what we get is just a warning.
Same issue as here. The warnings about the missing etc files break the JSON output. Adding empty files in the container fixes resizing.
Shouldn't this be reopened until the bug is fixed? Creating the two files also "solves" the issue for us.
/reopen cc
Reopened this issue. <detailsIn response to : /reopen cc Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
which version of ceph cluster is in use in your setup ?
We are running a ceph cluster with . kube-controller-manager is running which internally uses . Also just to prevent any confusion: I've used to resize the PVC, not rancher.
This also seems related to .
We are running
Are there plans to resolve this issue?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Answering myself - apparently it has been fixed in k8s 1.20:

Related discussion:
text: } if len(output) == 0 { <del> return 0, fmt.Errorf("can not get image size info %s: %s", b.Image, output) </del> <ins> return 0, fmt.Errorf("can not get image size info %s: %s", b.Image, string(output)) </ins> } <del> // Get the size value string, just between `size ` and ` MB in`, such as `size 1024 MB in 256 objects`. sizeIndex := strings.Index(output, imageSizeStr) divIndex := strings.Index(output, sizeDivStr) if sizeIndex == -1 || divIndex == -1 || divIndex <= sizeIndex+5 { return 0, fmt.Errorf("can not get image size info %s: %s", b.Image, output) } rbdSizeStr := output[sizeIndex+5 : divIndex] rbdSize, err := strconv.Atoi(rbdSizeStr) if err != nil { return 0, fmt.Errorf("can not convert size str: %s to int", rbdSizeStr) } </del> <ins> return getRbdImageSize(output) } </ins> <del> return rbdSize, nil </del> <ins> func getRbdImageSize(output []byte) (int, error) { info := struct { Size int64 `json:"size"` }{} if err := json.Unmarshal(output, &info); err != nil { return 0, fmt.Errorf("parse rbd info output failed: %s, %v", string(output), err) } return int(info.Size / rbdImageSizeUnitMiB), nil </ins> } // rbdStatus runs `rbd status` command to check if there is watcher on the image.