jiebi/CodeConvo huggingface dataset:
Code change:
timeout    time.Duration mu         sync.Mutex // protects active active     bool <del> // TODO: make this an net.IP address publicIP []string </del> <ins> publicIP   []string // TODO: make this a []net.IP </ins> } func (si *serviceInfo) isActive() bool {

Related discussion:
text: I'm not sure if it's the intended behaviour, but when I specify publicIPs for a service, external portals are only opened if createExternalLoadBalancer is set, too (this is checked in the OnUpdate function in ). After reading the docs, I would have expected that IPTables rules for external IPs are independent of the createExternalLoadBalancer flag. Also, we're running kubernetes locally, so setting createExternalLoadBalancer doesn't work since there's no cloud provider specified.
I was about to say it should work, but then I went and re-read the code and found at least one bug. Let me dig a bit more
You're right - the IPs actually propagate to the proxy, but it ignores them if createExternal... is not set. I think it is kosher to just do it. I'm prepping a PR
PR is up - if you feel bold you could test it in your environment...
Thanks, it's working fine with your patch.
Code change:
Now we will turn up two replicas of that image.  They all serve on port 8080, mapped to internal port 80 $ cd kubernetes <del> $ cluster/cloudcfg.sh -p 8080:80 run $DOCKER_USER/data 2 dataController </del> <ins> $ cluster/kubecfg.sh -p 8080:80 run $DOCKER_USER/data 2 dataController </ins> ### Step Three: Turn up the UX for the demo In a different terminal: $ cd kubernetes <del> $ cluster/cloudcfg.sh -proxy -www examples/update-demo/local/ </del> <ins> $ cluster/kubecfg.sh -proxy -www examples/update-demo/local/ </ins> Now visit the the [demo website](http://localhost:8001/static/index.html).  You should see two light blue squares with pod IDs and ip addresses.

Related discussion:
text: <!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: Which test(s) are failing: Overall Since when has it been failing: Nov 20 between 8:41 PST and 12:43 PST (it's been "failing" before that as well, but now none of the tests are running at all) Testgrid link: Reason for failure: Looks like  failed (), so is not building the env properly Slack discussion: Anything else we need to know*: /sig cluster-lifecycle /area provider/gcp
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
fixed here:
/assign
Code change:
Source string `json:"source,omitempty" description:"component reporting this event; short machine understandable string"` // Host name on which the event is generated. <del> Host string `json:"host,omitempty"` </del> <ins> Host string `json:"host,omitempty" description:"host name on which this event was generated"` </ins> // The time at which the client recorded the event. (Time of server receipt is in TypeMeta.) Timestamp util.Time `json:"timestamp,omitempty" description:"time at which the client recorded the event"`

Related discussion:
text: Or for some other reason it complains about v1beta1 and 2 on my system even though actually grepping those files shows no problems.
it's griping about Host
Code change:
RegisterSchedulable:                 true, ExperimentalKernelMemcgNotification: false, RemoteRuntimeEndpoint:               remoteRuntimeEndpoint, <del> RotateCertificates:                  false, </del> // TODO(#54161:v1.11.0): Remove --enable-custom-metrics flag, it is deprecated. EnableCustomMetrics: false, NodeLabels:          make(map[string]string),

Related discussion:
text: /kind cleanup What happened: and both require opt-in configuration knobs to the kubelet. They are kinda global settings, you probably either opt into dynamic cert rotation or you don't, so they qualify for being in the struct. The knob for enabling dynamic serving certs was (I fully support that :+1:). What you expected to happen: I'd like the same to happen to  to be consistent. (Also, we're planning to use the config file for the kubelet by default in kubeadm soon, and it'd be nice to be able to set this value then in a structured manner) /assign
Code change:
RegisterSchedulable:                 true, ExperimentalKernelMemcgNotification: false, RemoteRuntimeEndpoint:               remoteRuntimeEndpoint, <del> RotateCertificates:                  false, </del> // TODO(#54161:v1.11.0): Remove --enable-custom-metrics flag, it is deprecated. EnableCustomMetrics: false, NodeLabels:          make(map[string]string),

Related discussion:
text: Flag name: Help text: <Warning: Beta featureAuto rotate the kubelet client certificates by requesting new certificates from the kube-apiserver when the certificate expiration approaches. This is part of migrating the Kubelet command-line to a Kubernetes-style API. The flag should either be migrated to the Kubelet's API group, or simply removed from the Kubelet. If this could be considered an instance-specific flag, or a descriptor of local topology managed by the Kubelet, see: If this flag is only registered in os-specific builds, see: /assign /sig node /kind feature /priority important-soon /milestone v1.11 /status approved-for-milestone
[MILESTONENOTIFIER] Milestone Issue: Up-to-date for process <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. : New functionality. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
Code change:
<ins> /* Copyright 2024 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ // This is an empty file to avoid "build constraints exclude all Go files" error while running unit tests on non-linux platforms. package ipvs </ins>

Related discussion:
text: Tried running make test against local latest fork of kubernetes master Go version Go env Successfully able to run the test Try running make test on darwin os. No response <details</detailsLatest master <details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
Bro i think there might be problem with your setup , its not an issue .
test cases are working fine
Thanks for verifying, I will double check from my side, I hope you verified it on darwin os.
/sig release
Also seeing this when trying to run on mac
Code change:
package network import ( <ins> "encoding/hex" </ins> "encoding/json" "fmt" "math" <ins> "net" </ins> "strconv" "strings" "time" v1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" <ins> "k8s.io/apimachinery/pkg/util/wait" </ins> "k8s.io/kubernetes/test/e2e/framework" e2enode "k8s.io/kubernetes/test/e2e/framework/node" e2epod "k8s.io/kubernetes/test/e2e/framework/pod" <del> e2essh "k8s.io/kubernetes/test/e2e/framework/ssh" </del> "k8s.io/kubernetes/test/images/agnhost/net/nat" imageutils "k8s.io/kubernetes/test/utils/image"

Related discussion:
text: Which jobs are failing: kops-aws and many other jobs Which test(s) are failing: This isn't the actual test, but the text we're consistently seeing is Since when has it been failing: Testgrid links: (ref: ) Reason for failure: TBD Anything else we need to know: Very rarely tests pass when one of the tests passes: Seems related to , that was merged about the same time:
/sig cluster-lifecycle /sig aws
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig cluster-lifecycle /sig aws Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/priority critical-urgent this is merge blocking
/assign /cc /cc /cc
The PR we think we want to revert
is also failing, I don't think this is aws-specific
/cc
/assign
it failed with kind too :/ with the same message
Opened to revert
/unassign /sig network /sig testing /remove-sig cluster-lifecycle
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry. ‍
This may be the actual cause PR to address But we're going with revert first to be sure we've identified the offending PR
I think I've identified the root cause
Revert brought us back to green: Awaiting root cause fix
/remove-triage unresolved
/reopen merged but we're now seeing multiple tests fail since it merged Looking at it went red between these commits:
Reopened this issue. <detailsIn response to : /reopen merged but we're now seeing multiple tests fail since it merged Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
is an attempt to fix
reverting in
is the failure cluster I suspect will go down once the revert goes in
/close Failure cluster has gone down, calling this done <img width="998" alt="Screen Shot 2020-01-10 at 9 56 17 AM" src="https://user-"
Closing this issue.
Code change:
package network import ( <ins> "encoding/hex" </ins> "encoding/json" "fmt" "math" <ins> "net" </ins> "strconv" "strings" "time" v1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" <ins> "k8s.io/apimachinery/pkg/util/wait" </ins> "k8s.io/kubernetes/test/e2e/framework" e2enode "k8s.io/kubernetes/test/e2e/framework/node" e2epod "k8s.io/kubernetes/test/e2e/framework/pod" <del> e2essh "k8s.io/kubernetes/test/e2e/framework/ssh" </del> "k8s.io/kubernetes/test/images/agnhost/net/nat" imageutils "k8s.io/kubernetes/test/utils/image"

Related discussion:
text: <detailsIn response to : /close Failure cluster has gone down, calling this done <img width="998" alt="Screen Shot 2020-01-10 at 9 56 17 AM" src="https://user-"</details
Code change:
cpu: 500m # memory units are bytes memory: 64Mi <ins> requests: # cpu units are cores cpu: 500m # memory units are bytes memory: 64Mi </ins> ``` <del> The container will die due to OOM (out of memory) if it exceeds its specified limit, so specifying a value a little higher than expected generally improves reliability. </del> <ins> The container will die due to OOM (out of memory) if it exceeds its specified limit, so specifying a value a little higher than expected generally improves reliability. By specifying request, pod is guaranteed to be able to use that much of resource when needed. See [Resource QoS](../proposals/resource-qos.md) for the difference between resource limits and requests. </ins> If you’re not sure how much resources to request, you can first launch the application without specifying resources, and use [resource usage monitoring](monitoring.md) to determine appropriate values.

Related discussion:
text: did which addresses some of them, but turns up a lot more.
Alright, I will do that.
Code change:
testJSONPath( []jsonpathTest{ { <del> "when range is used in a certain way in script, additional line is printed", `{range .items[?(.status.phase=="Running")]}{.metadata.name}{" is Runningn"}`, </del> <ins> "range over pods without selecting the last one", `{range .items[?(.status.phase=="Running")]}{.metadata.name}{" is Runningn"}{end}`, </ins> data, "pod1 is Runningnpod2 is Runningnpod3 is Runningn", false, // expect no error

Related discussion:
text: no longer continues parsing an item when a key is encountered that has no value. This appears to be a behavior change introduced in Kubernetes version 1.22. will continue . Run the following commands against a cluster running CoreDNS pods in the namespace: Nothing. Kubernetes version (use ): v1.22.0 Cloud provider or hardware configuration: N/A OS (e.g: ): N/A Kernel (e.g. ): N/A Install tools: N/A Network plugin and version (if this is a network-related bug): N/A Others: N/A
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Debug data for Kubernetes version 1.21 cluster:
Debug data for Kubernetes version 1.22 cluster:
/sig cli
I reproduce the issue in my 1.23 environment. I'll try to fix it. /assign
PR works on it.
/unassign
to ensure I understand your comment correct, should fix this issue?
Yes, I've tested it. can resolve it. The bug is the line:
Yes, I've tested it. can resolve it. The bug is in this line:
Code change:
} } <ins> func (p *podWorkers) IsPodKnownTerminated(uid types.UID) bool { p.podLock.Lock() defer p.podLock.Unlock() if status, ok := p.podSyncStatuses[uid]; ok { return status.IsTerminated() } // if the pod is not known, we return false (pod worker is not aware of it) return false } </ins> func (p *podWorkers) CouldHaveRunningContainers(uid types.UID) bool { p.podLock.Lock() defer p.podLock.Unlock()

Related discussion:
text: There appears to be a change of behavior / possible regression in k8s 1.22 with respect to pod lifecycle behavior. We've started to see some cases when multiple pods are being created (often as part of a k8s job) the following will happen: pods get created by e.g. job controller (e.g. say 5). Each pod in job requests majority of node memory so only one such pod can fit on node at a time. schedules [first] pod to node, rest (4 pods) stay in pending accepts first pod, runs it to completion pod get scheduled to the node rejects the pod during pod admission, e.g. with , despite the fact that the first pod has already completed successfully This results in many of pods backing the job getting into Failed phase and having to be recreated by job controller. The job can often get into BackofLimit exceeded to hitting backoffLimit. The root issue seems to be a mismatch of when kubelet actually considers the pod to complete and what it reports to API server (which causes the followup pods to be scheduled to it). I'm still investigating further, but this may be related to the pod lifecycle changes in 1.22 -- I've created a e2e test as minimal repro of this behavior -- (Test fails) 22 build log -- 22 kubelet log -- (Test passes) 21 build log -- 21 kubelet log -- <!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --- Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.
Code change:
} } <ins> func (p *podWorkers) IsPodKnownTerminated(uid types.UID) bool { p.podLock.Lock() defer p.podLock.Unlock() if status, ok := p.podSyncStatuses[uid]; ok { return status.IsTerminated() } // if the pod is not known, we return false (pod worker is not aware of it) return false } </ins> func (p *podWorkers) CouldHaveRunningContainers(uid types.UID) bool { p.podLock.Lock() defer p.podLock.Unlock()

Related discussion:
text: </details
/sig node
/cc
Looking at the kubelet log in 1.22 () here's what I see -- First pod starts up... starts dies after ~10 seconds since the job is completed (the container simply runs a ) Pod completes, status is updated to API server Next pod comes in but is rejected with despite the earlier pod already being completed/terminated
did the kubelet pod lifecycle rework result in a reordering of kubelet resource bookeeping w.r.t. pod status? it seems reasonable for the scheduler to send another pod to a kubelet once it reports the pod complete and the use of the resources is done from an API perspective
I am also seeing this same behavior, but with "OutOfPods" rather than "OutOfMemory", when running a job with parallelism=100, completions=400 on GKE (which has a 110 pod per node limit).
this is easily reproducible on a local cluster as well start with this: then create this job: result on 1.22 is the job hangs with 2 completions and a bunch of rejections by the kubelet: on 1.21, the same job completes all pods successfully in a couple minutes with no OutOfpods errors
verified introduced the OutOfPods issue
Shame we didn’t have an e2e which caught this given how easy the reproducer is. This looks to be the opposite of the previous cpu admission bug (ignoring things that are terminating but not terminated). We are weak on admission testing in general. This potentially implies admission is looking at the wrong set of pods (admitted + terminating). Note there is already a bug in 1.21 and before where force deleted pods are incorrectly ignored from the set of allocated resources while they are terminating, but this doesn’t immediately seem to be that issue (since i doubt those jobs are being force deleted)
yeah, there was no force deletion involved in (or any deletion at all, actually... the completed pods just sat there, completed)
The set of admitted pods considered for pod count is everything in the pod manager filtered by filterOutTerminatingPods, which calls CouldHaveRunningContainers which returns true once a pod is seen by the workers (shortly after admission, and until we’ve confirmed via syncTerminatingPod that no containers are running). Whatever the problem is it’s likely in the interaction of that.
Code change:
} } <ins> func (p *podWorkers) IsPodKnownTerminated(uid types.UID) bool { p.podLock.Lock() defer p.podLock.Unlock() if status, ok := p.podSyncStatuses[uid]; ok { return status.IsTerminated() } // if the pod is not known, we return false (pod worker is not aware of it) return false } </ins> func (p *podWorkers) CouldHaveRunningContainers(uid types.UID) bool { p.podLock.Lock() defer p.podLock.Unlock()

Related discussion:
text: A pod the pod worker hasn’t seen yet is also filtered out (that means bursts of pods can get in, which seems the opposite of this problem, and is the bug i mentioned before).
Edit: one wrong thing spotted while looking - I'm not sure it fixes the issue you're seeing, but it's definitely wrong for admission (admission is filtering the known pod set down to remove pods that are terminated, but shouldn't filter pods that aren't admitted yet). This needs a node e2e reproducer that can run in serial (resource consumption is really hard to test from parallel, is slow in serial, and node e2e is back in a good state now) - / is that something either of you could help me with? I believe the reason the existing tests don't catch this is the job controller is creating pods fast enough that we see races in admission (multiple pods are admitted in one call to HandlePodAdditions) that are not caught in the current serial node e2e admission tests which rarely create more than one at a time.
Job pods still go through the scheduler to get assigned to a node, so even though the job controller can create pods in parallel, I'd still expect them to arrive at the node in an orderly fashion. It's still possible multiple arrive in a single HandlePodAdditions call, of course. It seems like kubelet admission is using laggy information that doesn't reflect pod completion the kubelet itself wrote to the pod API.
The bug turned out to be much simpler. Pods that are completed never go to the pod worker, so once a pod has reached terminal state it was being back to the list of active pods. The set of e2es that would have prevented this regression: node e2e that creates multiple pods at the same time and verifies that admission works correctly when HandlePodAdditions takes two pods at once node e2e that verifies that completed pods don't consume resources for admission purposes in the kubelet job controller e2e that attempts to verify some part of this reliably (ideally in a way that can be run in parallel e2e)
another related issue:
Linking here for completeness - a similar symptom might get exposed after this fix as described in The kubelet considers resources for terminating pods to be in use (they are!), but the scheduler ignores terminating pods and schedules new pods.
Code change:
} } <ins> func (p *podWorkers) IsPodKnownTerminated(uid types.UID) bool { p.podLock.Lock() defer p.podLock.Unlock() if status, ok := p.podSyncStatuses[uid]; ok { return status.IsTerminated() } // if the pod is not known, we return false (pod worker is not aware of it) return false } </ins> func (p *podWorkers) CouldHaveRunningContainers(uid types.UID) bool { p.podLock.Lock() defer p.podLock.Unlock()

Related discussion:
text: Because the kubelet now considers terminating pods, it rejects those rapidly rescheduled pods. The follow up is more involved but requires the kubelet to allow pods to be potentially admitted that can't be run currently due to those terminating pods. See the linked issue.
Code change:
) func TestCleanup(t *testing.T) { <ins> // The control plane is noisy and randomly logs through klog, for example: // E0912 07:08:46.100164   75466 controller.go:254] unable to sync kubernetes service: Endpoints "kubernetes" is invalid: subsets[0].addresses[0].ip: Invalid value: "127.0.0.1": may not be in the loopback range (127.0.0.0/8, ::1/128) // // By creating a ktesting logger and registering that as global // default logger we get the control plane output into the // "go test" output in case of a failure (useful for debugging!) // while keeping it out of the captured Ginkgo output that // the test is comparing below. // // There are some small drawbacks: // - The source code location for control plane log messages //   is shown as klog.go because klog does not properly //   skip its own helper functions. That's okay, normally //   ktesting should not be installed as logging backend like this. // - klog.Infof messages are printed with an extra newline. logger, _ := ktesting.NewTestContext(t) klog.SetLogger(logger) </ins> apiServer := testapiserver.StartAPITestServer(t) // This simulates how test/e2e uses the framework and how users

Related discussion:
text: pull-kubernetes-unit Probably since it was merged in The test checks output from the E2E framework and sometimes some extra log output shows up that isn't expected: No response /sig testing
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign