Change:
_, err = recyclerClient.CreatePod(pod) if err != nil { if errors.IsAlreadyExists(err) { <del> glog.V(5).Infof("old recycler pod %q found for volume", pod.Name) </del> <ins> deleteErr := recyclerClient.DeletePod(pod.Name, pod.Namespace) if deleteErr != nil { return fmt.Errorf("failed to delete old recycler pod %s/%s: %s", pod.Namespace, pod.Name, deleteErr) } // Recycler will try again and the old pod will be hopefuly deleted // at that time. return fmt.Errorf("old recycler pod found, will retry later") </ins> } else { return fmt.Errorf("unexpected error creating recycler pod:  %+vn", err) } } <del> defer func(pod *v1.Pod) { glog.V(2).Infof("deleting recycler pod %s/%s", pod.Namespace, pod.Name) if err := recyclerClient.DeletePod(pod.Name, pod.Namespace); err != nil { glog.Errorf("failed to delete recycler pod %s/%s: %v", pod.Namespace, pod.Name, err) } }(pod) </del> <ins> err = waitForPod(pod, recyclerClient, podCh) // In all cases delete the recycler pod and log its result. glog.V(2).Infof("deleting recycler pod %s/%s", pod.Namespace, pod.Name) deleteErr := recyclerClient.DeletePod(pod.Name, pod.Namespace) if deleteErr != nil { glog.Errorf("failed to delete recycler pod %s/%s: %v", pod.Namespace, pod.Name, err) } // Returning recycler error is preferred, the pod will be deleted again on // the next retry. if err != nil { return fmt.Errorf("failed to recycle volume: %s", err) } // Recycle succeeded but we failed to delete the recycler pod. Report it, // the controller will re-try recycling the PV again shortly. if deleteErr != nil { return fmt.Errorf("failed to delete recycler pod: %s", deleteErr) } return nil } </ins> <del> // Now only the old pod or the new pod run. Watch it until it finishes // and send all events on the pod to the PV </del> <ins> // waitForPod watches the pod it until it finishes and send all events on the // pod to the PV. func waitForPod(pod *v1.Pod, recyclerClient recyclerClient, podCh <-chan watch.Event) error { </ins> for { event, ok := <-podCh if !ok {

Discussion:
Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: If a pod is already running in the default namespace with the expected name (i.e. ), the PV recycle method just assumes it's a recycler pod. However, if it is not a recycler, it could result in the volume being reused without the data being cleared. What you expected to happen: The recycled volume should not be reused until the data has been verifiably deleted. The easiest approach would be to make it an error to not run the recycler pod successfully, i.e. delete these lines: I think that should be safe, since the recycler will retry. Other errors in that method should also be considered fatal, e.g.:
/cc
How about generating a unique name for each recycler pod so that it doesn't matter what existing pods are named?
Could someone from your team pick this up?
Ack
OpenShift runs the recycler pod in a dedicated namespace, not visible to users. Maybe it could be a good idea to do so also in Kubernetes. Anyway, killing the old recycler is probably the best option - it could be forged by user to block recycling and cheat quota. /assign
Agree with running in a dedicated namespace, but it sounds like this feature is just being deprecated, so it might not be worth the effort. Thanks for picking this up.