jiebi/CodeConvo huggingface dataset:
Code change:
} } <ins> func TestUpdatePodForTerminatedRuntimePod(t *testing.T) { podWorkers, processed := createPodWorkers() now := time.Now() podWorkers.podSyncStatuses[types.UID("1")] = &podSyncStatus{ startedTerminating: true, terminatedAt:       now.Add(-time.Second), terminatingAt:      now.Add(-2 * time.Second), gracePeriod:        1, } // creates synthetic pod podWorkers.UpdatePod(UpdatePodOptions{ UpdateType: kubetypes.SyncPodKill, RunningPod: &kubecontainer.Pod{ID: "1", Name: "1", Namespace: "test"}, }) drainAllWorkers(podWorkers) if len(processed) != 0 { t.Fatalf("Not all pods processed: %v", processed) } updates := processed["1"] if len(updates) != 0 { t.Fatalf("unexpected updates: %v", updates) } if len(podWorkers.lastUndeliveredWorkUpdate) != 0 { t.Fatalf("Unexpected undelivered work") } } </ins> func TestUpdatePodDoesNotForgetSyncPodKill(t *testing.T) { podWorkers, processed := createPodWorkers() numPods := 20

Related discussion:
text: The easy fixes will defiintely be "safe", but "slight regression in latency in a few rare cases". Serializing the pod worker caused this, but it's in theory rare.
My understanding is that we make pod worker the source of truth for pod state. In this kind of race condition, as long as the request is sent to the pod worker, and the pod UID matches, it should be able to reconcile the state. Even if we do this, it is not atomic, the race condition still exists in theory. We probably should make the pod worker be able to reconcile the state. I don't quite understand what you mean by "until the next housekeeping loop"? Since the pod is being terminated in , what problem would it cause if we just ignore the runningPod request? If the termination is still in progress, we need to wait for it to finish anyway.
Here's the most concerning scenario: is terminated and then deleted from apiserver kills the container is taking a very long time after 2, housekeeping fires and sees the stale cache, triggers a kill (proposed change) ignore the kill request (since terminated pod is running) restarts container in the pod that references a volume mount can't make progress because volume detach can't make progress because the restarted container is holding a device lock / has a file open / w/e pod worker for that pod retries forever is the minimal fix that ignores (5) but I'm thinking through how to handle 8. I don't think we should do the easy fix (refreshing the cache isn't enough because you are vulnerable to 6-8). I'm potentially considering having the pod worker invoke syncTerminatingPod in the event syncTerminatedPod exits with an error, or simply have syncTerminatedPod do it if it would return an error to address 8.
But this basically means that after , there could still be running containers. In that case, this wouldn't be the only problem, the whole assumption the refactor builds on becomes invalid? BTW, restarting container may happen to docker, but containerd doesn't support it (at least it can't be done easily), not sure about cri-o.
cri-o doesn't support restarting a container, either.
The assumption in the kubelet is there are no more containers started by the kubelet, which is indeed safe.
Code change:
} } <ins> func TestUpdatePodForTerminatedRuntimePod(t *testing.T) { podWorkers, processed := createPodWorkers() now := time.Now() podWorkers.podSyncStatuses[types.UID("1")] = &podSyncStatus{ startedTerminating: true, terminatedAt:       now.Add(-time.Second), terminatingAt:      now.Add(-2 * time.Second), gracePeriod:        1, } // creates synthetic pod podWorkers.UpdatePod(UpdatePodOptions{ UpdateType: kubetypes.SyncPodKill, RunningPod: &kubecontainer.Pod{ID: "1", Name: "1", Namespace: "test"}, }) drainAllWorkers(podWorkers) if len(processed) != 0 { t.Fatalf("Not all pods processed: %v", processed) } updates := processed["1"] if len(updates) != 0 { t.Fatalf("unexpected updates: %v", updates) } if len(podWorkers.lastUndeliveredWorkUpdate) != 0 { t.Fatalf("Unexpected undelivered work") } } </ins> func TestUpdatePodDoesNotForgetSyncPodKill(t *testing.T) { podWorkers, processed := createPodWorkers() numPods := 20

Related discussion:
text: That doesn’t prevent bugs in CRI implementations, user error, cosmic rays, basically computers. It’s unlikely and not something we really designed for before, but the prerefactor code would survive it. Appears to have gotten past the issue as expected. I’ll look at what it would take to mitigate the possibility of zombie containers such that the pod worker converges even if volume mount can’t be completed.
In general a little defensiveness here to unreliable external components has only minor cost, and some mitigation seems reasonable.
Opened the first part of the fix in
SG. Yeah, that should be considered a rare case. However, adding some defensive cleanup also SGTM as long as it doesn't introduce too much overhead or complexity. :)
Code change:
// create independent pod informer if required if setPodInformer { <del> podInformer = factory.NewPodInformer(context.clientSet, 12*time.Hour, v1.DefaultSchedulerName) </del> <ins> podInformer = factory.NewPodInformer(context.clientSet, 12*time.Hour) </ins> } else { podInformer = context.informerFactory.Core().V1().Pods() }

Related discussion:
text: <!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Scheduler's pod informer has a selector that . This is incorrect. Scheduler must watch all the pods of the cluster, including those scheduled by other schedulers in order to update its cache of the state of the cluster. Otherwise, it will miss important events and the amount of available resources, pods on nodes, ... which are needed for scheduler to work correctly, will not be updated properly. /sig scheduling /assign
cc
For updating it cache, shouldn't the default scheduler rely on node info/status? Also its not very clear, what information the default scheduler needs from other pods (not to be scheduled by it) that it makes decision on? Just trying to understand the issue more clearly.
Scheduler watches informer events to update its cache. Node status is not enough to update all the information that scheduler caches. Scheduler must know what pods are placed on what nodes. For example, in order to check inter-pod affinity/anti-affinity it needs to know the placement of all the pods, including those scheduled by other schedulers.
But this information is present here: So it seems that your PR will now make sure to get the scheduled pods' information directly from the pod informer.
So when I check the history, I see it was here:
have we seen any specific issues due to this change (like between default scheduler and DS controller)?
Yes, but scheduler relies on its pod informers to keep this location up to date. That's exactly why it is important for the scheduler to receive updates for all the pods in the cluster. Yes. We missed the problem at the time. This is problematic only in clusters that use multiple schedulers. I initially thought that DS controller could cause issues as well, but DS controller does not set , so it must not cause an issue. We have seen instances of scheduler's missing certain events in cluster and as a result scheduler's cache getting stale, but I doubt if it is related to this particular issue. This only affects clusters with more than one scheduler. If you run the test that I have written in with the head, it will fail. That tests simulate a multi-scheduler scenario.
Code change:
record.StartLogging(glog.Infof) } <del> func SetupEventSending(client *client.Client) { </del> <ins> func SetupEventSending(client *client.Client, hostname string) { </ins> glog.Infof("Sending events to api server.") <del> hostname := util.GetHostname("") </del> record.StartRecording(client.Events(""), api.EventSource{ Component: "kubelet",

Related discussion:
text: Running the pre-built 0.8.0 binaries this flag does not work. Looks like the
I think that code happens only when setting up events, right? The hostnameOverride flag seems to still be there: what do you think about making this function take hostnameOverride as a parameter?
Yeah, it should fix the issue. Will fix it tonight.
Code change:
// Ok, so we got something back from etcd. Let's set up a watch for new services, and // their endpoints <del> go s.WatchForChanges() </del> <ins> go util.Forever(s.WatchForChanges, 1*time.Second) </ins> for { services, endpoints, err = s.GetServices()

Related discussion:
text: kube-proxy regularly stops running with the below panic / stack-trace. This is when configured with vagrant. (It is also running on a laptop, so my unsupported-by-any-evidence theory is that it happens when the laptop resumes after sleep.) Restarting the service causes the proxy to continue working happily. (Should systemd be restarting the service?) Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: panic: runtime error: invalid memory address or nil pointer dereference Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: [signal 0xb code=0x1 addr=0x0 pc=0x431b51] Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: goroutine 22 [running]: Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: (0x6b7c00, 0xa51de8) Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /usr/local/go/src/pkg/runtime/panic.c:266 +0xb6 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: (0xc210066600, 0xc210037660, 0xc210037720, 0x77359400, 0x0) Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x841 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: (0xc210066600, 0xc210037660, 0xc210037720,
Code change:
// Ok, so we got something back from etcd. Let's set up a watch for new services, and // their endpoints <del> go s.WatchForChanges() </del> <ins> go util.Forever(s.WatchForChanges, 1*time.Second) </ins> for { services, endpoints, err = s.GetServices()

Related discussion:
text: 0x77359400) Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x165 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: created by Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x1b3 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: goroutine 1 [select (no cases)]: Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: () Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x47a Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: goroutine 3 [chan receive]: Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: (loggingT).flushDaemon(0xa57340) Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x50 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: created by ·1 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x276 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: goroutine 4 [syscall]: Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: () Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /usr/local/go/src/pkg/runtime/proc.c:1394 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: goroutine 6 [sleep]: Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: time.Sleep(0x12a05f200) Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: +0x31 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: (0x7aca18,
Code change:
// Ok, so we got something back from etcd. Let's set up a watch for new services, and // their endpoints <del> go s.WatchForChanges() </del> <ins> go util.Forever(s.WatchForChanges, 1*time.Second) </ins> for { services, endpoints, err = s.GetServices()

Related discussion:
text: 0x12a05f200) Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x57 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: created by Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x99 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: goroutine 7 [chan receive]: Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: (0xc210077120, 0x7ff2602627f0, 0xc210077060, 0xc210037540) Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x3a Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: created by Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x196 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: goroutine 8 [chan receive]: Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: (0xc210077210, 0x7ff260262840, 0xc210077150, 0xc2100375a0) Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x3a Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: created by Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x196 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: goroutine 9 [chan receive]: Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: (Mux).listen(0xc2100770c0, 0x718260, 0x4,
Code change:
// Ok, so we got something back from etcd. Let's set up a watch for new services, and // their endpoints <del> go s.WatchForChanges() </del> <ins> go util.Forever(s.WatchForChanges, 1*time.Second) </ins> for { services, endpoints, err = s.GetServices()

Related discussion:
text: 0xc210037600) Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x3a Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: ·001() Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x43 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: ·001() Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x40 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: (0xc21000a920, 0x0) Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x49 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: created by (*Mux).Channel Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x20a Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: goroutine 10 [chan receive]: Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: ·004() Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x4d Aug 13 12:04:06 kubernetes-minion-1 systemd[1]: kube-proxy.service: main process exited, code=exited, status=2/INVALIDARGUMENT Aug 13 12:04:06 kubernetes-minion-1 systemd[1]: Unit kube-proxy.service entered failed state.
I looked at this one a little bit and I think it's because of This line isn't present at , but I'm not sure which is newer. Presumably results in the channel being closed (and receiving a zero value that it then tries to dereference) on error.
Diagnostic: This line doesn't check for a closed channel, and it needs to. Yeah, how are we supposed to know if our client is going to close the channel or not?
Code change:
// Ok, so we got something back from etcd. Let's set up a watch for new services, and // their endpoints <del> go s.WatchForChanges() </del> <ins> go util.Forever(s.WatchForChanges, 1*time.Second) </ins> for { services, endpoints, err = s.GetServices()

Related discussion:
text: I think the behavior has changed at least once already. This is part of why I'm trying to (for our code, anyway) standardize all of our watches with a watch.Interface. I took the liberty of assigning this to you since you're more familiar with this code, but feel free to unassign if you can't take it right now.
Got it
can you give a try?
after i do the minion watch for kubelet config i'll come back and gut this and replace it.
Thanks - with merged I haven't seen this happening. Thanks for the super-quick fix!
Code change:
# Clean up kubectl delete namespace other <del> ############## </del> <ins> ########### </ins> # Secrets # <del> ############## </del> <ins> ########### </ins> ### Create a new namespace # Pre-condition: the test-secrets namespace does not exist

Related discussion:
text: The desciption of --all-namespaces is listing the requested object(s) across all namespaces. Namespace in current context is ignored even if specified with --namespace." But when I used the following command: error: namespace may not be empty when retrieving a resource by name no namespace set on resource pods "counter" The error messages are unclear, and the second message isn't labeled as an error even.
And why couldn't use --all-namespaces flag?
You're asking for a specific pod name (counter) in all namespaces, which is a nonsensical query. The pod name is not a grep-like filter, it's a specific name. Yeah, the errors could be better. Describe could work, but potentially generates a LOT of information and API calls.
But I found the following API is supported now. Get The result is: has the more information. Do we need to support the following kubectl command(using the above API) or just return clear error messages? get pods counter --all-namespaces
I'd be ok with this being implemented via the field selector.
The issue is still here :). What's our expected result? I'd like to contribute this.
, after check the code, the URL was built by ; do you mean we should update it; or avoid to use , instead, build url () directly.
After checking the code, it's better to add fieldSelector to , is created for fieldSelector. Will submit a PR based on .
getting an object with a particular name across all namespaces seems like something we shouldn't encourage... it breaks the assumption of namespace independence.
, so we'll only update message for this issue? for example:
Disagree. Need easy way to troubleshouting the problem "why pods is stuck in Pending status". It's tiring for every pod to find who created it and in which namespace. In the logs there is only the name of the pod.
Now, with field selectors we can do To further my understanding, I'd ask:- doesn't itself break independence? One can search by label across all namespaces, it feels funny for by name to be special.
Code change:
"k8s.io/apimachinery/pkg/util/uuid" clientset "k8s.io/client-go/kubernetes" coreclientset "k8s.io/client-go/kubernetes/typed/core/v1" "k8s.io/kubernetes/pkg/kubelet/util" "k8s.io/kubernetes/test/e2e/framework" e2epod "k8s.io/kubernetes/test/e2e/framework/pod" testutils "k8s.io/kubernetes/test/utils" <del> "github.com/onsi/ginkgo" "github.com/onsi/gomega" </del> ) var _ = framework.KubeDescribe("NodeProblemDetector [NodeFeature:NodeProblemDetector] [Serial]", func() {

Related discussion:
text: Which jobs are failing: Which test(s) are failing: Since when has it been failing: Last successful run (23 Oct 2020, 10:07 PDT): First unsuccessful run (23 Oct 2020, 12:08 PDT): Testgrid link: Reason for failure: Anything else we need to know: presubmit job in repository also started to fail in the same fashion. Attempted to debug it myself, found out the following stuff: last successful run was accessing the apiserver via HTTP: all unsuccessful runs attempted to access the apiserver via HTTPS but to no avail: perused the history of to find out that the PR is almost certainly the reason of the failures (or rather uncovered some problem with the NPD or its tests): first, it coincides with the start of the failures of , second, we changed the default hostname of the apiserver in node tests in that PR. Judging by the code and the changes in the aforementioned PR I think that a quick mitigation would be to add a flag to the of these jobs. /assign /sig node /cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Opened for a mitigation that would make those jobs stop failing.
This won't work, unfortunately. Copying response:
I have filed to address this issue, but it seems I could not run the test in advance to verify my patch, did you know how to manually trigger the presubmit job?
Unfortunately I don't :/. My PR was also a wild guess. could you take a look at the proposed fix? Thanks once again for your help - really appreciated :).
the submission of hasn't fixed the problems, unfortunately :/. Sample failing run: We still have some peculiar problem related to health checking: and now NPD panics, too:
Thanks for your report, I believe the reason is admission controller has been disabled in node e2e tests: We need to figure out a new approach now. /reopen
Reopened this issue.
Code change:
"k8s.io/apimachinery/pkg/util/uuid" clientset "k8s.io/client-go/kubernetes" coreclientset "k8s.io/client-go/kubernetes/typed/core/v1" "k8s.io/kubernetes/pkg/kubelet/util" "k8s.io/kubernetes/test/e2e/framework" e2epod "k8s.io/kubernetes/test/e2e/framework/pod" testutils "k8s.io/kubernetes/test/utils" <del> "github.com/onsi/ginkgo" "github.com/onsi/gomega" </del> ) var _ = framework.KubeDescribe("NodeProblemDetector [NodeFeature:NodeProblemDetector] [Serial]", func() {

Related discussion:
text: <detailsIn response to : Thanks for your report, I believe the reason is admission controller has been disabled in node e2e tests: We need to figure out a new approach now. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
filed . I hope we could completely address this issue this time.
/milestone v1.20
Code change:
<ins> // +build !windows /* Copyright 2018 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package dockershim import ( dockertypes "github.com/docker/docker/api/types" dockercontainer "github.com/docker/docker/api/types/container" runtimeapi "k8s.io/kubernetes/pkg/kubelet/apis/cri/runtime/v1alpha2" ) // applySandboxPlatformOptions applies platform specific options to dockercontainer.HostConfig and dockercontainer.ContainerCreateConfig. func (ds *dockerService) applySandboxPlatformOptions(hc *dockercontainer.HostConfig, config *runtimeapi.PodSandboxConfig, createConfig *dockertypes.ContainerCreateConfig, image string, separator rune) error { lc := config.GetLinux() if lc == nil { return nil } // Apply security context. if err := applySandboxSecurityContext(lc, createConfig.Config, hc, ds.network, separator); err != nil { return err } // Set sysctls. hc.Sysctls = lc.Sysctls return nil } </ins>

Related discussion:
text: <!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: Kubelet is depending on docker container's ResolvConfPath (e.g. ) to setup dns servers and search domains. While this is ok for Linux containers, ResolvConfPath is always an empty string for windows containers. So that the DNS setting for windows containers are always not set. For kubernetes clusters deployed by acs-engine, the Pod's DNS is always set to kube-dns service's clusterIP. This is done via the network plugin (e.g. l2bridge or azure-cni). It makes windows containers only work with ClusterFirst dns policy. To follow kubernetes dns policies, Kubelet should pass the expected DNS settings directly to windows containers (instead of set it in CNI plugins). What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig windows /sig node /kind bug /assign /cc
Let me fix this in v1.11
/priority important-soon
/status approved-for-milestone
This may have been fixed in
Not same. This is what has done, but it doesn't follow kubernetes dns policies.
status intended for 1.11?
Yep.
/status in-progress
[MILESTONENOTIFIER] Milestone Removed From Issue Important: Code freeze is in effect and only issues with may remain in the v1.11 milestone. <details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details