Change:
Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: create a pod with local volume scheduler start to do bind, it will make pv = controller will call syncVolume to make pv bound , then syncClaim will make pvc bound. But from the log, I find that after step 2, pv controller start to reclaim the pv because pvc not in cache. logs: We can see after pv reclaimed, then the logs related with pvc start to print out. I know kube-scheduler will wait for the pvc to be bound, then it will be scheduled again in this situation. But is there a better way to handle this situation at pv_controller? What you expected to happen: PV not reclaimed. How to reproduce it (as minimally and precisely as possible): Create pod and local volume pvc in large scale Anything else we need to know?: Environment: Kubernetes version (use ): k8s 1.9
/sig storage Maybe syncVolume could wait to make PV bound until PVC exists. wdyt?
/assign
/assign
Perhaps we cannot wait to make PV bound until PVC exists, because syncVolume needs to release PV if PVC does not exist. Even if scheduler pre-binds a PV, when its PVC is deleted, PV controller should release PV. The problem is that after binding a PV scheduler waits for PV controller to bind PVC, but PV controller may not have corresponding PVC (not synced or deleted), so it will release PV. PV controller pre-binds a PV the same way as scheduler, but PV/PVC binding processes share same cache, so it will not have problem. To fix this, we can check PVC existence in apiserver instead of local cache in at the cost of performance. Possible optimizing is we can add a specific annotation for scheduler, then can check PVC existence in apiserver only when volume is bound by scheduler.
Another optimizing is that we can get PVC from apiserver only if PVC does not exist in cache. Double-check PVC from apiserver to make sure PV is not reclaimed on API sync delay.
We are doing the same way at our prod env.
Fix with the second optimizing is much simpler. I created a PR:
Next time, you can submit a PR if you have a fix for issues you encountered ;-) By the way, how many nodes in your large scale cluster?
For this issue, I think there should be solution which no need to touch api-server to fix this issue. That's why I didn't submit the PR. Our cluster has more than 1k BM nodes, and we have more than 10K local volume PV. This issue happens when I create 1000 pods(which use local volume) with a for loop script. It is easy to reproduce.
This issue happens when cache is out of sync (perhaps heavy load on apiserver), that would be better if we can fix it without touching apiserver.
I am not sure I fully understand the issue. How is it possible that PV controller does not see the PVC in its cache? It must have been created before pod was created and scheduler bound it to a PV. That's IMO long enough for a pod to be in the cache, especially when PV got there quick enough.
The issue is syncing PVC/PV caches across scheduler and pv controller. The PVs have been created long ago. Now a bunch of PVCs and Pods are created with delayed binding. Now, the scheduler will process the pod -pvc, and then bind PV by setting PV.ClaimRef. However, the PV controller could be slow to get API updates, and does not know about the new PVC object yet, but it got the PV update from the scheduler. So it sees that PV is bound and pointing to a ClaimRef, but PVC doesn't exist, so it goes into Release state.

Discussion:
// the binding (PV->PVC or PVC->PV) was installed by the controller.  The // absence of this annotation means the binding was done by the user (i.e. // pre-bound). Value of this annotation does not matter. <ins> // External PV binders must bind PV the same way as PV controller, otherwise PV // controller may not handle it correctly. </ins> const annBoundByController = "pv.kubernetes.io/bound-by-controller" // This annotation is added to a PV that has been dynamically provisioned by

if err != nil { return err } <ins> if !found && metav1.HasAnnotation(volume.ObjectMeta, annBoundByController) { // If PV is bound by external PV binder (e.g. kube-scheduler), it's // possible on heavy load that corresponding PVC is not synced to // controller local cache yet. So we need to double-check PVC in //   1) informer cache //   2) apiserver if not found in informer cache // to make sure we will not reclaim a PV wrongly. // Note that only non-released and non-failed volumes will be // updated to Released state when PVC does not eixst. if volume.Status.Phase != v1.VolumeReleased && volume.Status.Phase != v1.VolumeFailed { obj, err = ctrl.claimLister.PersistentVolumeClaims(volume.Spec.ClaimRef.Namespace).Get(volume.Spec.ClaimRef.Name) if err != nil && !apierrs.IsNotFound(err) { return err } found = !apierrs.IsNotFound(err) if !found { obj, err = ctrl.kubeClient.CoreV1().PersistentVolumeClaims(volume.Spec.ClaimRef.Namespace).Get(volume.Spec.ClaimRef.Name, metav1.GetOptions{}) if err != nil && !apierrs.IsNotFound(err) { return err } found = !apierrs.IsNotFound(err) } } } </ins> if !found { glog.V(4).Infof("synchronizing PersistentVolume[%s]: claim %s not found", volume.Name, claimrefToClaimKey(volume.Spec.ClaimRef)) // Fall through with claim = nil