Change:
/kind bug What happened: kubelet logs lot of when it starts and a pod was deleted while it was offline. How to reproduce it (as minimally and precisely as possible): a pod with AWS EBS volume (should work also with GCE or any other volume) kubelet the pod kubelet It's not always reproducible, is there some sort of race? My current theory: desiredStateOfWorldPopulator scans pods and adds this pod to DSW reconciler.syncStates() runs. it finds mounted volume but it does not add it to actual state of world because it's in DSW: When kubelet fully starts, it kills the pod and cleans DSW, but the volume never reached AWS so nothing is unmounted. Environment: Kubernetes version (use ): today's master I have troubles reproducing this on master. /sig storage /sig node
Ok, check at won't add the volume to ASW. But the volume is in DSW and subsequent reconcile() should check that the volume is attached and mounted and add it to ASW. This does not happen, because: Volume is not in because it was overwritten by new kubelet (?)
This problem goes beyond a simple logging error. The problem also is - such volumes never get unmounted correctly and hence volume can not be used anywhere. IMO this is kind of serious. Steps to reproduce: a pod on node X kubelet the pod. The pod will be stuck in "Terminating" state back the kubelet There is a race which is rather easy to reproduce and what happens is - volume gets to DSW at kubelet but it never makes into ASW because volume is indeed never in use on node (because pod was deleted while kubelet was down). This results in volume never getting unmounted and hence any attempt to use the volume on different node fails.
IMO, this happens: is called only once at kubelet startup and at that point the pod still exists (it's in Terminating state). The volume is in DSW and it's not to ASW. is called periodically. It could add the volume to ASW, but it does not, because the volume is not in yet. kills pod containers and confirms it's deleted. The pod is deleted from API server. removes the pod from DSW is called and the volume is not there because it's not in DSW. We have a mounted volume that's not in DSW nor in ASW. There are several possible solutions: a. kubelet can add the volume to ASW in regardless of DSW. b. kubelet can update node status earlier, just after . Will there still be race when pod is deleted during and thus it won't reach ? c. kubelet could call regularly. I am inclined to a), it looks robust to me and it just reflects reality in ASW. It should not matter if the volume is in DSW or not at this point - will sort it out and do the right thing.
solution d.: in reconciler.syncStates() try to reconstruct volume from DSW, fall back to only if the volume is not there. DSW already has spec and other useful info.
Experimental fix (solution d.):
FYI, this might be another 1.10 (or 1.10.1) candidate. We need thorough review though, volumeManager is quite complicated.
another simpler option is perhaps to mark a volume as "ReportedInUse" if it was found on the disk during reconstruction phase. That will make sure that, volume gets correctly to ASW via reconciler(rather than us doing it on reconstruction) and it will make sure that volume is unmounted when volume does get removed from DSW.
I closed , it would blow up when kubelet was restarted at wrong time when creating a pod. Interesting idea, I opened with it. I am not sure it covers all the corner cases though. Newly started kubelet does not overwrite its own status until runs, I hope it's ensured by something and not just a coincidence.
I think might still hit a race condition issue as below is normally deleted (without force delete) during kubelet restart restarts, pod is received again so it is add to the pod Manager is to desired state, so reconciler skip to reconstruct to the actual state kill containers and then try to delete the Pod from API server. If actual state has not yet updated from reconciler, pod will be deleted from pod manager and desired state too will reconcile actual state and desired state. But if step 4 happened first, it will not be able to update actual state since volume is removed from desired state. Depending on which happens first between step 4 and 5, volume might or might not have a chance to get into the actual state. I am thinking seems better to solve the issue with some additional protection. The idea is to avoid letting kubelet to delete pod before actual state has updated the volume. We can set a flag in actual state to indicate when reconstruction finishes.
what hook you are planning to use to stop kubelet from deleting pod unless it has been processed into asw?
My thought is to add a flag (e.g., finishReconstruct) in actualstateofworld to indicate whether reconstruction finished or not. volumemanager will add a function (e.g., IsActualStateReconstructed) to check the flag is set or not. When call canBeDeleted() function, the PodResourcesAreReclaimed in will return false if actual state has not yet finished reconstructing.
Talked with offline, come up with a solution which combines both and . The proposal still follows the idea of pod is to desired state (no matter whether it might be removed soon or not), do not reconstruct directly (reconstruction does not has the full information which might fail or cause problem having wrong information since there is not enough checking), instead reconcile should have a chance to reconcile actual state by going through the volume set up process. if the pod was force deleted from API server during kubelet down, reconciler tries to clean up mounts directly if possible. The main problem currently is that pod might be deleted from desired state before reconciler updates the actual state. is trying to prevent of this. makes sure that reconciler can update the actual state as soon as possible.
please add this to 1.10 milestone. This should be a 1.10 blocker.
SIG-leads, please confirm, thanks!
/priority critical-urgent /status in-progress /kind bug
You must be a member of the github team to add status labels.
Adding to 1.10 as blocker per
/status in-progress
[MILESTONENOTIFIER] Milestone Issue: Up-to-date for process Note: This issue is marked as , and must be updated every 1 day during code freeze. Example update: <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
I spoke with offline. The volume reconstruction work (PR ) is non-critical but touches critical mount/unmount code paths. While the PRs that and have ( and ) may resolve the issue identified here, I'm afraid there may be other regressions introduced that we are not aware of. Therefore, I'm asking Jing to revert the volume reconstruction PR. We can reintroduce it along with the patches early in 1.11 along with more extensive testing. Unfortunately, because changes went in on top, an automatic revert is not possible. Jing is working on a manual revert. could you please help test her revert PR when it is ready?
Update on this issue: spent the morning trying to create a revert PR , but it turns out a couple of other big refactors (including ) went in on top of her volume reconstruction PR, making it much more difficult to revert. At this point says she can either spend more time backing out her changes and, in doing so, possibly introduce new bugs or spend time making sure the proposed fixes are rock solid. She says she feels confident that the bugs are understood, and the proposed PRs fixing it should be sufficient, and would like to spend time to double and triple checking it with instead. So and will proceed with this.
i support this approach
I support this as well. Thank you and for working on this.
okay I integrated both the PRs in a branch and tested both on AWS and GCE . Ran e2e tests as well and it looks all good.
I lied about e2e test run. A second run surfaced some failures:
the error message indicates the pods failed to enter running condition. I think my fix should not affect pod creation.
yeah I agree. I noticed that these tests use NFS based PVs which are not ideal source for testing the bug we are trying to reproduce. In this case the NFS server itself may not start correctly and hence the test may fail. I am looking into refactoring the test suite, so as it can use GCE/EBS sources as well.
I created a follow up issue for creating the disruptive e2e tests so that bugs like these can be more easily caught - . Will follow up with PR tomorrow. Also ran e2e test after removing NFS part and just relying on GCE-PD and it all passed. I am going to let run couple of hours in a loop and see if there is a regression.
thanks for your work here. How did it come out?
we still have some problems that needs to be addressed for . I will sync up with when online.
Perfect, thank you.
/reopen
you can't re-open an issue/PR unless you authored it or you are assigned to it. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
can we keep it open, we still need the second PR
/cc
re-opening per
The second PR got merged late night yesterday as well. So we can close this. /close

Discussion:
return } volumesNeedUpdate := make(map[v1.UniqueVolumeName]*reconstructedVolume) <ins> volumeNeedReport := []v1.UniqueVolumeName{} </ins> for _, volume := range podVolumes { <del> if rc.desiredStateOfWorld.VolumeExistsWithSpecName(volume.podName, volume.volumeSpecName) { glog.V(4).Infof("Volume exists in desired state (volume.SpecName %s, pod.UID %s), skip cleaning up mounts", volume.volumeSpecName, volume.podName) continue } </del> if rc.actualStateOfWorld.VolumeExistsWithSpecName(volume.podName, volume.volumeSpecName) { glog.V(4).Infof("Volume exists in actual state (volume.SpecName %s, pod.UID %s), skip cleaning up mounts", volume.volumeSpecName, volume.podName) <ins> // There is nothing to reconstruct </ins> continue } <ins> volumeInDSW := rc.desiredStateOfWorld.VolumeExistsWithSpecName(volume.podName, volume.volumeSpecName) </ins> reconstructedVolume, err := rc.reconstructVolume(volume) if err != nil { <ins> if volumeInDSW { // Some pod needs the volume, don't clean it up and hope that // reconcile() calls SetUp and reconstructs the volume in ASW. glog.V(4).Infof("Volume exists in desired state (volume.SpecName %s, pod.UID %s), skip cleaning up mounts", volume.volumeSpecName, volume.podName) continue } // No pod needs the volume. </ins> glog.Warningf("Could not construct volume information, cleanup the mounts. (pod.UID %s, volume.SpecName %s): %v", volume.podName, volume.volumeSpecName, err) rc.cleanupMounts(volume) continue } <ins> if volumeInDSW { // Some pod needs the volume. And it exists on disk. Some previous // kubelet must have created the directory, therefore it must have // reported the volume as in use. Mark the volume as in use also in // this new kubelet so reconcile() calls SetUp and re-mounts the // volume if it's necessary. volumeNeedReport = append(volumeNeedReport, reconstructedVolume.volumeName) glog.V(4).Infof("Volume exists in desired state (volume.SpecName %s, pod.UID %s), marking as InUse", volume.volumeSpecName, volume.podName) continue } // There is no pod that uses the volume. </ins> if rc.operationExecutor.IsOperationPending(reconstructedVolume.volumeName, nestedpendingoperations.EmptyUniquePodName) { glog.Warning("Volume is in pending operation, skip cleaning up mounts") }

glog.Errorf("Error occurred during reconstruct volume from disk: %v", err) } } <ins> if len(volumeNeedReport) > 0 { rc.desiredStateOfWorld.MarkVolumesReportedInUse(volumeNeedReport) } </ins> } func (rc *reconciler) cleanupMounts(volume podVolume) {