jiebi/CodeConvo huggingface dataset:
Change:
w.resultsManager.Set(w.containerID, result, w.pod) <ins> if w.probeType == liveness && result == results.Failure { // The container fails a liveness check, it will need to be restared. // Stop probing until we see a new container ID. This is to reduce the // chance of hitting #21751, where running `docker exec` when a // container is being stopped may lead to corrupted container state. w.onHold = true } </ins> return true }

Discussion:
Happened in GKE suite:
Does it need a liveness probe, maybe?
to delegate
Saw
Can you take a look at this one? Thanks!
FWIW, I saw some fishy messages in in (node: 104.197.48.84) It seems like docker was giving conflicting view of the container status, i.e., whether the container was running or not (and whether kubelet should kill it). kubelet's first attempt to kill the container : All the exec well after the grace period still shows that the container is the "Running: true" state () Later, kubelet tried killing the container again and failed because container does not exist. Note that after the failed kill also shows that the container was in the Running state. At the same time, cadvisor observed the container being stopped but then detected the container and then tried adding it again, but failed to find the /cc UPDATE: According to supervisord, docker was not killed during this time.
I didn't find result for the first Killing, does that really return? What is the log level, I remember that you told me before the log level in GKE is 2?
AFAICT, the pod worker may have never returned because the specific pod worker didn't log anything beyond that point. I was looking at (docker stop hangs) to see if it's relevant. didn't print any error though.
Yeah, the last SyncPod log was which is before the first Killing. Then the pod worker was stuck in
This really looks similar with I tried to reproduce it following the comment. The container was not actually killed. Docker stop hangs. Docker ps shows the container is still running. But the difference is that when I tried to kill it with another , it still hung there.
I was able to reproduce this quite easily by running 30 pods with the same spec in the test and with :
From
My theory is that during may cause docker to leave the container in a unrecoverable state (yep, a bug). I can consistently reproduce this by simply creating (which always fails), and leave the cluster for < 10 min. The restart counts of some of the pods will stop increasing after sometime, and after ssh'ing in to the node, one can easily confirm the same symptoms posted above. If pods use the http probe, they do not have this issue.

For the workaround, we can instruct the probe workers to stop probing after detecting a liveness check failure, until the container has been restarted (with a new container ID). Note that this is just a workaround, as it doesn't stop other components or users to run during this window. I've verified that with this fix, there are no zombie docker containers anymore. Will upload a PR.
FWIW, I also tested it with docker 1.8.3, which has the same problem.
Thanks !
Change:
COPY cni-conf /etc/cni/net.d # Create symlinks for each hyperkube server <del> # TODO: this is unreliable for now (e.g. running "/kubelet" panics) # Also, it doesn't work for other architectures </del> <ins> # TODO: replace manual symlink creation with --make-symlink command once # cross-building with qemu supports go binaries. See #28702 </ins> # RUN /hyperkube --make-symlinks <ins> RUN ln -s /hyperkube /apiserver  && ln -s /hyperkube /controller-manager  && ln -s /hyperkube /federation-apiserver  && ln -s /hyperkube /federation-controller-manager  && ln -s /hyperkube /kubectl  && ln -s /hyperkube /kubelet  && ln -s /hyperkube /proxy  && ln -s /hyperkube /scheduler </ins>

Discussion:
The command was removed from the dockerfile in this PR: This addition was originally proposed here: I'd like to fix the problem or re-enable the creation of symlinks if none exists. Currently there is not enough information from the PR that removed them to know what the problem is. I've tested the symlink command and successfully have built hyperkube images with it enabled. Calling the kubelet symlink doesn't panic for me. Do you know how to reproduce this issue?
You're right, the panic I observed when giving no arguments is gone with v1.3.0 It doesn't work when cross-building Qemu doesn't play very well with Go binaries Can't we just for all binaries in the hyperkube image? I can make a patch with that if you like
It was discussed in that the idea with the command was there to obviate the need to maintain all the commands in the dockerfile since the hyperkube binary already knows all the registered servers. Do you have more information about why this doesn't work with cross-building? I don't quite understand why Go binaries not playing nicely with qemu matter, since, at some point hyperkube is full of a bunch of go binaries that have to play nicely with arm.
Qemu can't run Go binaries, that's the problem. We can use virtually anywhere else Until Qemu releases a version that's compatible with Go, we can't run Go binaries when cross-building
Ooh I understand now, basically any use of go during cross building will not work. Do you happen to know if this more a Go issue or qemu issue? Which upstream project needs to make a fix?
It's a qemu issue, talked to the golang folks Some qemu maintainer said they've improved stuff at HEAD, but haven't tested it Anyway, we have to wait until the next qemu release. If we want it now, we initially have to go with
Change:
if err != nil { return errors.Wrap(err, "unable to fetch CoreDNS current installed version and ConfigMap.") } <ins> var hasCoreDNSMigrationFailed bool </ins> if IsCoreDNSConfigMapMigrationRequired(corefile) { if err := migrateCoreDNSCorefile(client, coreDNSConfigMap, corefile, currentInstalledCoreDNSVersion); err != nil { <del> return err </del> <ins> // Errors in Corefile Migration is verified during preflight checks. This part will be executed when a user has chosen // to ignore preflight check errors. hasCoreDNSMigrationFailed = true klog.Warningf("the CoreDNS Configuration was not migrated: %v. The existing CoreDNS Corefile configuration has been retained.", err) if err := apiclient.CreateOrRetainConfigMap(client, coreDNSConfigMap, kubeadmconstants.CoreDNSConfigMap); err != nil { return err } </ins> } } else { if err := apiclient.CreateOrUpdateConfigMap(client, coreDNSConfigMap); err != nil {

Discussion:
What happened: Upgrade using kubeadm failed because of CoreDNS migration enchancement What you expected to happen: Upgrade passed correctly How to reproduce it (as minimally and precisely as possible): any 1.15.x Kubernetes with CoreDNS that deployed with any unusual image (self built for example). We using commits counter as part of image tag, e.g - 1.6.2-100. to upgrade to 1.16.x with kubeadm. Anything else we need to know?: Logs part without ignoring preflight: Logs part with ignoring preflight: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OpenStack OS (e.g: ):Kernel (e.g. ):Install tools: Kubeadm
/sig cluster-lifecycle
The error came from It seems you should mention this issue on coredns project
True. Will notify them also.
/cc
/assign In such cases where the migration isn't supported and the user chooses to ignore the warnings from the preflight check, the migration should be skipped. wdyt? /cc
Yes. IMO, the warnings should be phrased like "CoreDNS configuration migration will not occur: reason"... so that a user expects the config migration will not be done.
/area kubeadm
SGTM
Upgrade passed correctly I think the best we can do here is to allow the upgrade to pass, but this will, in most practical cases, result in a cluster without a functioning DNS. The most common practical use-case for using a self built image of CoreDNS is to use external plugins. During the kubeadm upgrade, CoreDNS will be replaced with a stock release version of CoreDNS. When that new version of CoreDNS starts, it will not be able to load the existing Corefile's external plugins, and exit. ... or, we could finish the upgrade, leaving the existing CoreDNS deployment untouched, which I think works better.
Yea, it's kinda strong words that "Upgrade passed correctly". Probably, I think the best solution will be to make possible skip Corefile migration and make kubeadm finish the upgrade without Corefile changes. Guys that using custom images of CoreDNS should care about Corefile for new version by themselves, IMO.
Yes, I think that would work best...
Change:
kubecontainer "k8s.io/kubernetes/pkg/kubelet/container" "k8s.io/kubernetes/pkg/kubelet/network" utilexec "k8s.io/kubernetes/pkg/util/exec" <ins> "k8s.io/kubernetes/pkg/util/wait" </ins> ) const (

Discussion:
Host network pods should be able to be created before the container network exists. Pod syncs for pods on the container network should fail until the container network is configured. cc
ref:
I think this is a thing we should make working before v1.4, so adding the milestone. Complain if you disagree. Also, does volunteer to make a PR? :)
Going to make a first pass at this in
Great!
/cc
Change:
// enabled. func TestPreferNominatedNode(t *testing.T) { defer featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.PreferNominatedNode, true)() <del> testCtx := initTestPreferNominatedNode(t, "perfer-nominated-node") t.Cleanup(func() { testutils.CleanupTest(t, testCtx) }) cs := testCtx.ClientSet defaultPodRes := &v1.ResourceRequirements{Requests: v1.ResourceList{ v1.ResourceCPU:    *resource.NewMilliQuantity(100, resource.DecimalSI), v1.ResourceMemory: *resource.NewQuantity(100, resource.DecimalSI)}, } </del> defaultNodeRes := map[v1.ResourceName]string{ v1.ResourcePods:   "32", v1.ResourceCPU:    "500m", v1.ResourceMemory: "500", } <del> type nodeConfig struct { name string res  map[v1.ResourceName]string </del> <ins> defaultPodRes := &v1.ResourceRequirements{Requests: v1.ResourceList{ v1.ResourceCPU:    *resource.NewMilliQuantity(100, resource.DecimalSI), v1.ResourceMemory: *resource.NewQuantity(100, resource.DecimalSI)}, </ins> } tests := []struct { name         string <del> nodes        []*nodeConfig </del> <ins> nodeNames    []string </ins> existingPods []*v1.Pod pod          *v1.Pod runnningNode string }{ { <del> name: "nominated node released all resource, preemptor is scheduled to the nominated node", nodes: []*nodeConfig{ {name: "node-1", res: defaultNodeRes}, {name: "node-2", res: defaultNodeRes}, }, </del> <ins> name:      "nominated node released all resource, preemptor is scheduled to the nominated node", nodeNames: []string{"node-1", "node-2"}, </ins> existingPods: []*v1.Pod{ initPausePod(&pausePodConfig{ Name:      "low-pod1", <del> Namespace: testCtx.NS.Name, </del> Priority:  &lowPriority, NodeName:  "node-2", Resources: defaultPodRes, }), }, pod: initPausePod(&pausePodConfig{ <del> Name:      "preemptor-pod", Namespace: testCtx.NS.Name, Priority:  &highPriority, </del> <ins> Name:     "preemptor-pod", Priority: &highPriority, </ins> Resources: &v1.ResourceRequirements{Requests: v1.ResourceList{ v1.ResourceCPU:    *resource.NewMilliQuantity(500, resource.DecimalSI), v1.ResourceMemory: *resource.NewQuantity(200, resource.DecimalSI)},

Discussion:
<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Integration tests are sometimes flaking locally: PreferNominatedNode Once the node gets deleted, the nodelifecycle controller is racing to update pod status and the pod deletion logic is failing causing tests to flake. Ultimately, we call testCtx.CleanUp() which cleans the nodes and the pods involved the test namespace so we are not going to leak any resources. links to appreciated links to specific failures in spyglass appreciated <!-- You can identify the SIG from the "prowjobconfigurl" on the testgrid dashboard for a test --/sig scheduling <!-- Please see the deflaking doc () for more guidance! --/assign
cc Tried addressing it via
/triage accepted thanks!
Change:
return diskUUID, nil } <ins> // GetHost returns host of the virtual machine func (vm *VirtualMachine) GetHost(ctx context.Context) (mo.HostSystem, error) { host, err := vm.HostSystem(ctx) var hostSystemMo mo.HostSystem if err != nil { klog.Errorf("Failed to get host system for VM: %q. err: %+v", vm.InventoryPath, err) return hostSystemMo, err } s := object.NewSearchIndex(vm.Client()) err = s.Properties(ctx, host.Reference(), []string{"summary"}, &hostSystemMo) if err != nil { klog.Errorf("Failed to retrieve datastores for host: %+v. err: %+v", host, err) return hostSystemMo, err } return hostSystemMo, nil } </ins> // DetachDisk detaches the disk specified by vmDiskPath func (vm *VirtualMachine) DetachDisk(ctx context.Context, vmDiskPath string) error { device, err := vm.getVirtualDeviceByPath(ctx, vmDiskPath)

Discussion:
Currently whenever a vSphere volume is provisioned, we are looking at all nodes in the cluster and then fetching their hosts and then their datastores and then ensuring that selected datastore in present on all the nodes. This is done every time a new PVC is provisioned. It can be pretty expensive in large clusters and number of involved API calls can be big. /sig storage /area vsphere cc
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : Currently whenever a vSphere volume is provisioned, we are looking at all nodes in the cluster and then fetching their hosts and then their datastores and then ensuring that selected datastore in present on all the nodes. This is done every time a new PVC is provisioned. /area vsphere Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted
/assign
Change:
// Volume binder only wants to keep unassigned pods sched.VolumeBinder.DeletePodBindings(pod) } <ins> sched.Framework.RejectWaitingPod(pod.UID) </ins> } func (sched *Scheduler) addPodToCache(obj interface{}) {

Discussion:
What would you like to be : When the pod is scheduled to the phase and I delete the pod, the pod still remains in the until the plugin allow/reject/timeout(Especially when waiting for a long time). Why is this needed: We need to take the initiative to delete the data in the and terminate the process in waiting state of plugin to ensure data accuracy, if the pod has been deleted /cc /priority important-soon /assign /sig scheduling
I think we can update the event handler for deleting pods to reject the pod. Can you send a PR for that?
OKI will send a pr for it.
Change:
c, _ := storage.Delete(svc.ID) <-c <del> if len(fakeCloud.Calls) != 1 || fakeCloud.Calls[0] != "delete" { </del> <ins> if len(fakeCloud.Calls) != 2 || fakeCloud.Calls[0] != "get-zone" || fakeCloud.Calls[1] != "delete" { </ins> t.Errorf("Unexpected call(s): %#v", fakeCloud.Calls) } srv, err := memory.GetService(svc.ID)

Discussion:
pull-kubernetes-e2e-azure-disk-windows pull-kubernetes-e2e-azure-disk-windows It seems caused by connection failed: from kubelet logs: This repeats until timeout
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig windows
/area providers/azure
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /area providers/azure Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Last run (): The run before (): no code changes in between
Now my PR passed the lane without any code changes (just yet another retest):
note that is not a presubmit for merge, or else it would show: <img width="89" alt="Screen Shot 2021-03-08 at 11 30 57 PM" src="https://user-"non-required presubmits can be bypassed with , but are generally worth looking at first
/assign Can you take a look?
There's something odd going on here. As the test description indicates, it deletes the pod for the StatefulSet and expects it to be recreated. When the kubelet attempts to recreate the pod, however, the mount point for the volume is no longer there. From the log above, we can see the pod was deleted succesfully. It was then rescheduled but failed because the volume mount point does not exist. In the kubelet logs, we see the successful mount for the initial pod creation: We then see the delete of the pod and the failed attempt to recreate it: Finally during cleanup, we see a failure during unmount the volume because the mount point does not exist: None of the other collected logs show what happened to the original mount point.
Change:
if err != nil { klog.Error(log("registrationHandler.RegisterPlugin failed at CSI.NodeGetInfo: %v", err)) if unregErr := unregisterDriver(pluginName); unregErr != nil { <del> klog.Error(log("registrationHandler.RegisterPlugin failed to unregister plugin due to previous: %v", unregErr)) </del> <ins> klog.Error(log("registrationHandler.RegisterPlugin failed to unregister plugin due to previous error: %v", unregErr)) </ins> return unregErr } return err

Discussion:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: If kubelet plugin registration fails during the registration process, the kubelet never attempts to register the socket again (assuming the error was transient), unless the socket for the driver is deleted & recreated to trigger a new registration. What you expected to happen: Maybe retry with exponential backoff when kubelet plugin registration fails instead of just logging and waiting for another filesystem event like it does today. Of course, if a socket returns a terminal error (does not implement the kubelet plugin registration service, for example), we should not retry. Follow up on /milestone v1.14 /sig storage How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
I'd like to work on this issue
/assign
Re opening. was a temporary work around for this. is the permanent fix.
Hello! We are starting Bug Freeze for 1.15. Is this issue still planned on being incorporated into 1.15?
the dependent PR is not yet merged. can we get this in for 1.15?
said that is the "permanent fix." and that has merged. So let's close this out /close
Closing this issue. <detailsIn response to : said that is the "permanent fix." and that has merged. So let's close this out Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Change:
}}, metav1.CreateOptions{}) framework.ExpectNoError(err, "failed to create pod") framework.Logf("created %v", podTestName) <ins> framework.ExpectNoError(e2epod.WaitForPodNameRunningInNamespace(f.ClientSet, podTestName, f.Namespace.Name)) framework.Logf("running and ready %v", podTestName) </ins> } // wait as required for all 3 pods to be found

Discussion:
<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI -- [sig-node] Pods should delete a collection of pods [Conformance] TBD links to appreciated links to specific failures in spyglass appreciated <!-- Please see the deflaking doc () for more guidance! --
cc /sig windows
I've looked through the logs for a bit and the test itself. What the test is doing: create 3 pods. expect to find 3 pods (it doesn't seem to wait for the to start, they only need to exist). delete all 3 of them as a collection. expect all 3 pods to dissapear. All 3 pods ended up on the same node, plus some other tests were running in parallel, which could also contribute to the failure. Looking at the kubelet logs, it seems that was the remaining pod to be deleted. Filtering the logs, we can see: The test failed at, while the other 2 pods did: Those times correspond with the number of pods observed by the test: Now, my question is, why are there such big gaps between the ` events? It's 12s for test-pod-2 and 21s for test-pod-3? Seems like a long time for pods / containers to linger after they've been killed. PS: It seems that Containers are still being created / started / stopped, even if the delete request came less than 0.2s after their creation. It doesn't seem to me like a reasonable thing to do, especially if that Pod would have to pull a huge image; I would have expected to be able to cancel that.
/triage accepted
Let's discuss at the 8/17 SIG-Windows community meeting
Hit again on 8/15
I think was planning on opening up a PR to use longer timeouts in this test. We see that some Windows containers do not get scheduled until 1+ minutes into the test - especially when there are many tests running in parallel.
Change:
} if err != nil { if ctx.Err() != nil { <del> Failf("Controller %s: Failed to Get from replica %d [%s]: %vn pod status: %#v", r.controllerName, i+1, pod.Name, err, pod.Status) return false, err </del> <ins> // We may encounter errors here because of a race between the pod readiness and apiserver // proxy. So, we log the error and retry if this occurs. Logf("Controller %s: Failed to Get from replica %d [%s]: %vn pod status: %#v", r.controllerName, i+1, pod.Name, err, pod.Status) return false, nil </ins> } Logf("Controller %s: Failed to GET from replica %d [%s]: %vnpod status: %#v", r.controllerName, i+1, pod.Name, err, pod.Status) continue

Discussion:
https://k8s- Failed: [] Network Partition [Disruptive] [Slow] [] [ReplicationController] should recreate pods scheduled on the unreachable node AND allow scheduling of pods on a node after it rejoins the cluster {Kubernetes e2e suite}
Builds: ci-kubernetes-e2e-gci-gke-serial ci-kubernetes-e2e-gke-serial-release-1.5 ci-kubernetes-e2e-gke-serial Failed: [] Network Partition [Disruptive] [Slow] [] [ReplicationController] should recreate pods scheduled on the unreachable node AND allow scheduling of pods on a node after it rejoins the cluster {Kubernetes e2e suite}
Builds: ci-kubernetes-e2e-gci-gke-serial-release-1.5 ci-kubernetes-e2e-gke-containervm-1.4-gci-1.5-upgrade-cluster-new ci-kubernetes-e2e-gke-serial-release-1.5 Failed: [] Network Partition [Disruptive] [Slow] [] [ReplicationController] should recreate pods scheduled on the unreachable node AND allow scheduling of pods on a node after it rejoins the cluster {Kubernetes e2e suite}
this is currently assigned to you -- can you take a look? I guess it has not failed individually recently, only as part of large "broken test runs" so maybe we should close it? Please advise.
This has only failed once in the last couple of weeks in https://k8s- and zero times in https://k8s- so I think we should close it. WDYT?
Yes, fixed a regression in this recently. Closing this.
It seems like this test is still flaking a lot on GKE
should this block 1.6?
Re-opening until we've verified it's fixed. It seems to have just failed again
I think the serial tests haven't picked up the fix yet.
Tests are looking good, closing if I see another success.
We need to cherrypick the fix in into 1.6. I've filed a separate issue to track the hanging network calls on the GKE apiserver. /cc
I believe this was fast forwarded onto the branch by
-- can we take this off of the 1.6 release now? It seems to be a test setup issue
Yes, this is not a blocker anymore.

this can be removed from the 1.6 milestone
This Issue hasn't been active in 49 days. It will be closed in 40 days (Jun 14, 2017). cc You can add 'keep-open' label to prevent this from happening, or add a comment to keep it open another 90 days