jiebi/CodeConvo huggingface dataset:
Code change:
pullQPS float32 // Optional, maximum burst QPS from the docker registry, must be positive if QPS is > 0.0 pullBurst int <ins> // Optional, no statistics will be available if omitted cadvisorClient CadvisorInterface cadvisorLock   sync.RWMutex } // SetCadvisorClient sets the cadvisor client in a thread-safe way. func (kl *Kubelet) SetCadvisorClient(c CadvisorInterface) { kl.cadvisorLock.Lock() defer kl.cadvisorLock.Unlock() kl.cadvisorClient = c } // GetCadvisorClient gets the cadvisor client. func (kl *Kubelet) GetCadvisorClient() CadvisorInterface { kl.cadvisorLock.RLock() defer kl.cadvisorLock.RUnlock() return kl.cadvisorClient </ins> } // Run starts the kubelet reacting to config updates

Related discussion:
text: So today, cadvisor became wedged and just stopped paying attention to its socket. Not kubelet's fault. But when I would launch all I would get would be: It took me a bit of trolling through netstat and lsof to figure out that the problem was cadvisor being completely locked up. kill -9 cadvisor and everything started working well. I wonder if we should emit more messages during startup about what operation we are about to attempt, not just the results of operations, as I was blaming docker for a while....
kubelet shouldn't block on this. I'll take a look.
Any ideas what was locking cAdvisor? and was it just not responding to requests?
I don't have any idea why it was locked. If I find another one locked I'll hang onto it to debug. Journal had: obviously the 'kill' came from me. It would not respond to systemctl cadvisor stop. netstat looking at port 4194 showed me At which point I realized cadvisor wasn't handling its sockets right since so many long dead kubelet sockets hadn't been cleaned up, so I kill -9'd it.
I cut some interesting info from the cadvisor journal log (trying to cut hostnames) You can see the last message about failed to update stats for container ... happened Oct 08 01:06:01. It was then silent until I killed it Oct 08 19:20:13, even though I'm pretty sure dnf was still being launched every hour on the hour. So it was wedged for about 18 hours.
What kernel is this on? We've seen there is a kernel bug fixed in 3.12+ that sometimes hangs us forever on exit. We're not quite sure how it gets triggered...
16.3
Code change:
Bind(ctx context.Context, binding *v1.Binding, opts metav1.CreateOptions) error Evict(ctx context.Context, eviction *policy.Eviction) error GetLogs(name string, opts *v1.PodLogOptions) *restclient.Request <ins> ProxyGet(scheme, name, port, path string, params map[string]string) restclient.ResponseWrapper </ins> } // Bind applies the provided binding to the named pod in the current namespace (binding.Namespace is ignored).

Related discussion:
text: Currently services/proxy is available in client-go, but pods/proxy resource is not. This requires manual usage via the rest client like This can be seen in Kubernetes e2e tests as well
/kind feature /sig api-machinery
Code change:
if newRS != nil && newRS.UID == rsUID { continue } <del> if len(podList.Items) > 0 { return true </del> <ins> for _, pod := range podList.Items { switch pod.Status.Phase { case v1.PodFailed, v1.PodSucceeded: // Don't count pods in terminal state. continue case v1.PodUnknown: // This happens in situation like when the node is temporarily disconnected from the cluster. // If we can't be sure that the pod is not running, we have to count it. return true default: // Pod is not in terminal phase. return true } </ins> } } return false

Related discussion:
text: Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: The deployment controller for a deployment with a single replica failed to create a new ReplicaSet when the old ReplicaSet included an Evicted pod. Typically, the eviction was caused by "node low on resource: nodefs". When we manually deleted the evicted pod, the new RS and accompanying pod were created immediately. We have seen this happen at least a dozen times. Here are the audit logs for a recent case: ! What you expected to happen: The deployment controller ignores the evicted pod and creates the updated replicaset/pod. How to reproduce it (as minimally and precisely as possible): To reproduce, you would need a way to trigger a kubelet-driven eviction on a node that is currently running a single-replica Deployment with . Wait for the pod to be evicted and the new pod to be created, confirm that the evicted pod still exists, and then make a trivial change to the deployment spec. The running pod will be deleted, and no new replicaset/pod will be created. This issue does not seem to occur for manual evictions (via ); pods I evict that way disappear from the API server immediately. Anything else we need to know?: Possibly related to and/or Environment: Kubernetes version (use ): v1.8.5-gke.0 Cloud provider or hardware configuration: GKE OS (e.g. from /etc/os-release): Container-Optimized OS
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Thanks for the detailed bug report!
Code change:
if newRS != nil && newRS.UID == rsUID { continue } <del> if len(podList.Items) > 0 { return true </del> <ins> for _, pod := range podList.Items { switch pod.Status.Phase { case v1.PodFailed, v1.PodSucceeded: // Don't count pods in terminal state. continue case v1.PodUnknown: // This happens in situation like when the node is temporarily disconnected from the cluster. // If we can't be sure that the pod is not running, we have to count it. return true default: // Pod is not in terminal phase. return true } </ins> } } return false

Related discussion:
text: From Here's what happened: Deployment was updated by the user, so it scaled all old ReplicaSets to 0 old ReplicaSet was scaled to 0 (ReplicaSet's == 0), but still owned the evicted pod, which wouldn't be killed automatically because of the above comment (ReplicaSet's 0) Deployment found that there were still old Pods, and kept waiting for them to be gone ~(wait for old ReplicaSet's to be 0)~ (Edit: Deployment also checks those old Pods by listing them to make sure there are none, not just by checking ReplicaSet's status) before starting the new rollout. We can either modify Deployment controller (to ignore evicted old pods) or ReplicaSet controller (to kill evicted pods ~when it gets scaled down~), if this is not the intended behavior. WDYT?
I am actually debugging likely the same production issue with Recreate strategy in two clusters right now. (We run on patched 1.9.1.) I have traced it down to broken RS as well. in 1 cluster RS leaves around pod failed matching the selector in the other one those are evicted pods. all replicas are scaled down to 0 and report 0 in status but pods with valid controllerRefs to them are still present. Same is 0 for me but pod is still there I think the correct behaviour is for replicaSet scaled to 0 to delete all of its pod, not to leave them laying around
xref - this should fix it
Code change:
BalanceAttachedNodeVolumes:                  {Default: false, PreRelease: utilfeature.Alpha}, DynamicProvisioningScheduling:               {Default: false, PreRelease: utilfeature.Alpha}, VolumeSubpathEnvExpansion:                   {Default: false, PreRelease: utilfeature.Alpha}, <ins> KubeletPluginsWatcher:                       {Default: false, PreRelease: utilfeature.Alpha}, </ins> // inherited features from generic apiserver, relisted here to get a conflict if it is changed // unintentionally on either side:

Related discussion:
text: <!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: PR introduced a new kubelet plugin registration mechanism. It is not flag gated. What you expected to happen: Since this feature is alpha, it should be flag gated and disabled by default (opt-in). How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: CC
[MILESTONENOTIFIER] Milestone Issue Needs Attention Action required: During code slush, issues in the milestone should be in progress. If this issue is not being actively worked on, please remove it from the milestone. If it is being worked on, please add the label so it can be tracked with other in-flight issues. Note: This issue is marked as , and must be updated every 3 days during code slush. Example update: <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
Code change:
return } } <del> // Listing pods shouldn't really fail, as we are just querying the informer cache. selector, err := metav1.LabelSelectorAsSelector(jobObj.Spec.Selector) if err != nil { utilruntime.HandleError(fmt.Errorf("parsing deleted job selector: %v", err)) return } pods, _ := jm.podStore.Pods(jobObj.Namespace).List(selector) for _, pod := range pods { if metav1.IsControlledBy(pod, jobObj) && hasJobTrackingFinalizer(pod) { jm.enqueueOrphanPod(pod) } } </del> <ins> jm.backupRemovePodFinalizers(jobObj) </ins> } // enqueueSyncJobImmediately tells the Job controller to invoke syncJob

Related discussion:
text: When a Job is deleted (via Background cascade), we have a backup logic to remove finalizers from pods This logic could also be applied for any finished Job. While investigating a case similar to , we observed that the job controller didn't delete finalizers from Pods where the Job was deleted with Foreground cascading policy. Since the Job was already marked as Finished, it only makes sense to trigger the backup logic to remove finalizers, so that we can guard against any possible bug where a Job is marked as Finished but not all pod finalizers are removed. Note that I'm not aware of any other bug in tracking with finalizers.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig apps
/remove-kind feature /kind cleanup
/wg batch
I am happy to give this a go.
/assign
Just to check if I have the logic chain right: We are talking about adding this logic If so, I should probably turn the backup-logic code into a function that then apply that in both places? I also would be unsure of where in the unit tests this would be tested, are you able to point me to how this is tested in the deletePod function? I assume we add a pod that is never included in the "uncounted" lists.
I would put it in the event handler, to be more of a "catch all" Yes, that's a good idea. I think you would have to force it quite a bit, as we have fixed all the bugs that we have found. For example, from the test: the controller, but do not start the workers (this will setup the event handlers) a job the pods the job the workers some inspiration here
Alright, thank you :))
Should the issue title be: Remove finalizers from Pods belonging to finished Jobs ?
If someone has set a custom finalizer on a Pod, even one owned by a Job, (when) would we remove that finalizer?
/retitle Remove tracking finalizers from Pods belonging to finished Jobs No, only the finalizer specific to job tracking
Code change:
message ResourceSliceList { // Standard list metadata // +optional <del> optional .k8s.io.apimachinery.pkg.apis.meta.v1.ListMeta listMeta = 1; </del> <ins> optional .k8s.io.apimachinery.pkg.apis.meta.v1.ListMeta metadata = 1; </ins> // Items is the list of resource ResourceSlices. repeated ResourceSlice items = 2;

Related discussion:
text: Listing s on v1.31.0 returns an object with a field instead of the standard field for (list) objects: Listing s on v1.30.3 behaves correctly: This is also propagated to the Swagger spec (which is how I noticed it, since I maintain the Rust bindings for the client API). v1.31.0: v1.30.3: As explained above. v1.31.0 with feature gate enabled and runtime config . With kind, this is: the API server request as in the above curl command in any way you like as long as it lets you see the raw JSON response. No response <details</details<detailskind 0.23.0 </details<details</details<details</details<details</details<details</details
/sig api-machinery
It seems there may be a discrepancy in the API response format between v1.30.3 and v1.31.0. The object is expected to follow standard Kubernetes API conventions, which typically use the field for list objects.
This deviation might cause compatibility issues for API clients that rely on this structure.
If this deviation from the expected API structure wasn't intentional, it should be aligned with the standard conventions.
The presence of instead of could disrupt client integrations, as you've encountered with the Swagger spec and Rust bindings.
cc
/triage accepted
/assign
this must be fixed to json:"metadata" protobuf:"bytes,1,opt,name=metadata"`` the serialization will not work with standard clients
/priority critical-urgent
Shoot, that's clearly a mistake.
Code change:
package gce import ( <ins> "flag" </ins> "fmt" <ins> "net" </ins> "net/http" "sort" "strconv"

Related discussion:
text: From GCE email today: Current behavior: You are not required to add firewall rules for Network LB health checks today Action required: We are introducing three new ranges for health checking: 209.85.152.0/22, 209.85.204.0/22, and 35.191.0.0/16. You need to ensure that your firewall allows traffic from 209.85.152.0/22, 209.85.204.0/22, and 35.191.0.0/16. The health check probes to your backend instances will come from one of these ranges. We have until Jan 31, 2018 - 1 year.
/assign this issue explains the confusion I had several weeks ago when speaking with and you about L4 health checks. I didn't realize a firewall rule isn't currently required.
/assign
Is this done now?
Not yet, there is a WIP on .
/close
Code change:
kl.sourcesReady.AddSource(u.Source) case e := <-plegCh: <ins> if e.Type == pleg.ContainerStarted { // record the most recent time we observed a container start for this pod. // this lets us selectively invalidate the runtimeCache when processing a delete for this pod // to make sure we don't miss handling graceful termination for containers we reported as having started. kl.lastContainerStartedTime.Add(e.ID, time.Now()) } </ins> if isSyncPodWorthy(e) { // PLEG event for a pod; sync it. if pod, ok := kl.podManager.GetPodByUID(e.ID); ok {

Related discussion:
text: These tests have flaked consistently since merged: in many cases, they are the only failure in a test run: Either the functionality introduced by that PR is not reliable, or the test is faulty. /sig node /priority critical-urgent /assign
/milestone v1.19
/assign
Looks like this is failing ~5% of pull-kubernetes-node-e2e runs
FYI
Looks like there might still be a race somewhere looking at the kubelet logs Success case: Fail case:
Looking at it more, it seems like is unexpectedly empty here Success: Fail:
Stacks of interest I think we might be dealing with a stale If there is a call before the container is to the Pod in the runtime and the is called within the 2 second default cache period, it seems possible that the in the could have no containers.
what do you think of this analysis? If so, any obvious solutions occur to you?
If I'm correct, didn't introduce this; just the e2e test exposed an existing bug in the kubelet.
That all looks correct. I'm taking a look at the proposed solutions
Code change:
TimeoutSeconds:      5, // default 1s can be pretty aggressive in CI environments with low resources FailureThreshold:    1, } <del> pod := gRPCServerPodSpec(nil, livenessProbe, "etcd") </del> <ins> pod := gRPCServerPodSpec(nil, livenessProbe, "agnhost") </ins> RunLivenessTest(ctx, f, pod, 1, defaultObservationTimeout) })

Related discussion:
text: Tests in are using image for gRPC probe tests: should not be restarted with a GRPC liveness probe should be restarted with a GRPC liveness probe We should use image for these tests as it is much smaller size. /sig node /kind cleanup /priority important-soon /area tests /help-wanted /good-first-issue KEP:
This request has been marked as suitable for new contributors. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : Tests in are using image for gRPC probe tests: /kind cleanup /priority important-soon /area tests /help-wanted /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : Tests in are using image for gRPC probe tests: /kind cleanup /priority important-soon /area tests /help-wanted /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hi can I work on this issue? I'm new to Kubernetes and want to contribute!
Should I change to in this file?
Yes.
Code change:
TimeoutSeconds:      5, // default 1s can be pretty aggressive in CI environments with low resources FailureThreshold:    1, } <del> pod := gRPCServerPodSpec(nil, livenessProbe, "etcd") </del> <ins> pod := gRPCServerPodSpec(nil, livenessProbe, "agnhost") </ins> RunLivenessTest(ctx, f, pod, 1, defaultObservationTimeout) })

Related discussion:
text: Also pass proper parameters to agnhost to enable gRPC endpoint, change ports, and run tests to ensure that this worked.
Thanks for your reply! Will let you know if I have more questions as I work
/assign
/assign
Code change:
Write-Output $c throw "Contents of /etc/secret/foo.txt are not as expected" } <del> if ($env:NODE_NAME_TEST -ne $env:COMPUTERNAME) { throw "NODE_NAME_TEST env var ($env:NODE_NAME_TEST) does not equal COMPUTERNAME ($env:COMPUTERNAME)" </del> <ins> # Windows ComputerNames cannot exceed 15 characters, which means that the $env:COMPUTERNAME # can only be a substring of $env:NODE_NAME_TEST. We compare it with the hostname instead. # The following comparison is case insensitive. if ($env:NODE_NAME_TEST -ine "$(hostname)") { throw "NODE_NAME_TEST env var ($env:NODE_NAME_TEST) does not equal hostname" </ins> } Write-Output "SUCCESS"` )

Related discussion:
text: Windows GCE jobs: [sig-windows] [Feature:WindowsHostProcessContainers] [MinimumKubeletVersion:1.22] HostProcess containers should run as a process on the host/node [sig-windows] [Feature:WindowsHostProcessContainers] [MinimumKubeletVersion:1.22] HostProcess containers should support various volume mount types Since the jobs were reintroduced. Windows COMPUTERNAMEs are typically limited to 15 characters [1], while the kubelet nodes spawned in the GCE test jobs are much longer (e.g.: `). The names might be truncated, resulting in the test failure. [1] No response /sig windows /sig testing
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
dupe of /close
Closing this issue. <detailsIn response to : dupe of /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Code change:
Write-Output $c throw "Contents of /etc/secret/foo.txt are not as expected" } <del> if ($env:NODE_NAME_TEST -ne $env:COMPUTERNAME) { throw "NODE_NAME_TEST env var ($env:NODE_NAME_TEST) does not equal COMPUTERNAME ($env:COMPUTERNAME)" </del> <ins> # Windows ComputerNames cannot exceed 15 characters, which means that the $env:COMPUTERNAME # can only be a substring of $env:NODE_NAME_TEST. We compare it with the hostname instead. # The following comparison is case insensitive. if ($env:NODE_NAME_TEST -ine "$(hostname)") { throw "NODE_NAME_TEST env var ($env:NODE_NAME_TEST) does not equal hostname" </ins> } Write-Output "SUCCESS"` )

Related discussion:
text: I want to run the e2e test but it failed. Reason: Node name and hostname is longer than 15 characters. But the ComputerName only used the first 15 characters. So they don't equal and the test failed. The e2etest should pass regardless of the hostname length. Deploy a Windows with long nodename (longer than 15 characters). Then, run e2e test. No response <details</details<details</details<details</details<details</details<details</details<details</details
/sig testing
Can you show the error message?
STEP: Destroying namespace "host-process-test-windows-2574" for this suite. • Failure [6.334 seconds] [sig-windows] [Feature:WindowsHostProcessContainers] [Excluded:WindowsDocker] [MinimumKubeletVersion:1.22] HostProcess containers should run as a process on the host/node [It] Nov 21 22:12:49.003: Expected <v1.PodPhase: Failed to equal <v1.PodPhase: Succeeded {"msg":"FAILED [sig-windows] [Feature:WindowsHostProcessContainers] [Excluded:WindowsDocker] [MinimumKubeletVersion:1.22] HostProcess containers should run as a process on the host/node","total":1,"completed":0,"skipped":1147,"failed":1,"failures":["[sig-windows] [Feature:WindowsHostProcessContainers] [Excluded:WindowsDocker] [MinimumKubeletVersion:1.22] HostProcess containers should run as a process on the host/node"]}
/sig windows does this hostname error ring a bell?
We discussed this a bit on slack, the test is using %COMPUTERNAME% to validate we are running on the host. This value is truncated to 15 chars on windows due to netbios. We could use the hostname, which isn't truncated, with a fallback to computer name for this validation to make the test validate more scenarios. Important to note that some scenarios might not work if in older environments that use netbios but for our test purposes this will work. Long discussion on netbios here:
We'll be working on this as part of a broader windows operational readiness work. Thanks wenli for digging into it.