jiebi/CodeConvo huggingface dataset:
Code change:
Operator: v1.TolerationOpExists, Effect:   v1.TaintEffectNoSchedule, }, <ins> { Key:      algorithm.TaintNodeNotReady, Operator: v1.TolerationOpExists, Effect:   v1.TaintEffectNoSchedule, }, </ins> }, }, }

Related discussion:
text: Is this a BUG REPORT or FEATURE REQUEST?: What happened: admission control plugin in apiserver. a deployment with 1 replica. The pod is successfully launched. By default the admission control plugin will add following toleration in the pod. the node where the above pod is running. of image locality policy and above toleration in the pod, scheduler still choose the node to launch the pod. The pod is in state and failed to create. What you expected to happen: The pod should be scheduled to healthy node and created successfully. How to reproduce it (as minimally and precisely as possible): See above steps. Anything else we need to know?: Environment: Kubernetes version (use ): v1.10 Cloud provider or hardware configuration: openstack OS (e.g. from /etc/os-release): Ubuntu 16.04 Kernel (e.g. ): 4.4.0-121-generic Install tools: N/A Others:
/sig scheduler
/sig scheduling /sig node
I think that we should avoid scheduling pods to nodes which is not ready even it can tolerate and , comments
As the pod already claimed to tolerate NotReady, this is an expected behavior. P.S. If the eviction dose not happen after 5m, that should be the bug. Please reopen the issue and modify the description. This is what tolerations is designed for :) And there are many use cases for pod to be scheduled to NotReady nodes (mostly during cluster deployment).
I think maybe you can configure effect to avoid scheduling to those nodes.
scheduler still choose the NotReady node to launch the pod For running, the toleration is used for graceful termination by ; if is enabled, it's better to also taint node with for .
Wondering how the pod can start to run if the Node is not ready at deployment time? Can you specify a use case here?
It won't need to start, system pod pending will be expected :) The point is, freedom for users to decide what tolerations they want matters here. I don't think it's reasonable to combine with in this case.
Perhaps, add to default toleration of admission control plugin could eliminate the confusion.
We should align the behavior when enabled or disabled : if is disabled, will not schedule pod to not-ready node. if is enabled, we should also taint node with .
Code change:
Operator: v1.TolerationOpExists, Effect:   v1.TaintEffectNoSchedule, }, <ins> { Key:      algorithm.TaintNodeNotReady, Operator: v1.TolerationOpExists, Effect:   v1.TaintEffectNoSchedule, }, </ins> }, }, }

Related discussion:
text: with timeout is used for graceful termination of 'running pods' for feature. We should not re-schedule pod to not-ready unless tolerating it explicitly.
reopen this for more discussion :)
Let me extracted the mixed problems in this issue :-) First one: These two default tolerations attached to the pod by means: if the node is tainted as or , wait for 300s before the pod is evicted from the node. While according to the issue description, is working as expected (pods are not evicted immediately). This part should be closed. Second one: The discussion raised is actually about taint during eviction: if TaintNodesByCondition is enabled, we should also taint node with NotReady:NoSchedule. My thoughts is: it's reasonable to taint node with as well for consistency consideration (but I'm hesitate if only for eliminating confusion as pods are expected to pending on NotReady nodes) I think it should be clearer now.
I open a PR If TaintNodesByCondition is enable, taint node with NotReady:NoSchedule and Unreachable:NoSchedule.
, we did not catch 1.11 release, so this issue will be handled in 1.12 :)
in 1.9.7enable the TaintNodesByCondition and taint nodee.g. node01 with NotReady:NoSchedulethen pod also can not be scheduled to node01 even when node01 is healthy. What you expected to happen: The pod should be scheduled to node01 when it is healthy node and not be scheduled to node01 when it is not ready. Thanks
How did you do that? by kubectl? The fix is not ready until 1.12.
YesI tried "kubectl taint nodes node01 ",and also tried "kubectl taint nodes node01 "
this feature did not handle taint manually
Ok, thanks a lot! So I can not solve the problem manually now and what I should do is just waiting for the version 1.12, right?
Code change:
t.Errorf("expected %d ProxyLoops running, got %d", want, got) } <ins> func waitForNumProxyClients(t *testing.T, s *serviceInfo, want int, timeout time.Duration) { var got int now := time.Now() deadline := now.Add(timeout) for time.Now().Before(deadline) { s.activeClients.mu.Lock() got = len(s.activeClients.clients) s.activeClients.mu.Unlock() if got == want { return } time.Sleep(500 * time.Millisecond) } t.Errorf("expected %d ProxyClients live, got %d", want, got) } </ins> func TestTCPProxy(t *testing.T) { lb := NewLoadBalancerRR() service := proxy.ServicePortName{NamespacedName: types.NamespacedName{Namespace: "testnamespace", Name: "echo"}, Port: "p"}

Related discussion:
text: Depends on UDP clients have a timeout (so we don't leak connections forever), but the test does not cover this. To do this, we probably need to track active connections, and plumb that all the way down to the goroutines spawned by the ProxyLoop routines. This is probably something we want to do anyway to be able to monitor.
Code change:
// Reference to this node. nodeRef *v1.ObjectReference <del> // Track initialization initialized bool </del> // imageCache is the cache of latest image list. imageCache imageCache

Related discussion:
text: when kubelet restart,all images which notused cleaned. but there is a flag in kubelet,,explainMinimum age for an unused image before it is garbage collected. Examples: '300ms', '10s' or '2h45m'. i set to ,it's means if a image not used,it will live 20m and then clean by kubeletif has disk pressure.if kubelet not restart, it work,but if restart ther is a bug, not work,image will be cleaneven if it's no enough time. work, image even if not used,it will live 20m and then clean by kubelet. 1make node trigger disk pressure. 2pull some image which not used by pod 3restart kubelet 4image cleaned by kubelet after around 10s i see the source code, and maybe there is a bug. in func,well detect image and update firstdetect time for every image. and in func,well check images livetime by firstDetected. if kubelet running normal,it work,because there is a func. in this line,,in first time,ts is zero. so image's firstDetected time is zero,and in freeSpace this check is always false,so image will be cleaned. maybe we can change to ,can fix this bug. <details</details<detailsNone </details<details</details<details</details<details</details<details</details
/sig node
/assign
/triage accepted /priority backlog initial assessment of priority, subject to change
Code change:
# Deepcopy # local input_pkgs=() <del> while read -r file; do dir="$(dirname "${file}")" </del> <ins> while read -r dir; do </ins> pkg="$(cd "${dir}" && GO111MODULE=on go list -find .)" input_pkgs+=("${pkg}") done < <( <del> ( kube::codegen::internal::git_grep -l  </del> <ins> ( kube::codegen::internal::git_grep -l --null  </ins> -e '+k8s:deepcopy-gen='  ":(glob)${root}"/'**/*.go'  || true  <del> ) | LC_ALL=C sort -u </del> <ins> ) | xargs -0 -n1 dirname  | LC_ALL=C sort -u </ins> ) if [ "${#input_pkgs[@]}" != 0 ]; then

Related discussion:
text: function in script has following issues: function only supports file but we have multiple types file with name as in . Hence this function is not working in our case. For the mitigation there is already an open . I have asked for including our changes as well. We have multiple files under a for example and . Because of this for appears twice and it generates duplicate informers We only need unique elements before passing it to . I followed this approach for unique elements: We have packages under . Earlier we were generating client only for and . But with the new I am not able to do that. I am using the following call to function: It results in following flags to As there is no directory as , it doesnot generate anything. Is it still possible to call on specific folder as it was done with earlier scripts? One mitigation I can think of is to call function on whole and then remove the marker from the types for which genclient is not needed. But the issue is we won't be able to pass the flag for each pkg. function in script is not working as expected.
/sig api-machinery
/cc
/triage accepted
Hi . I knew that there were all sorts of clients out there doing all sorts of things with these scripts, so I am not surprised to find you. :) Do you have a PR which attempts to use kube_codegen, so I can repro your issues directly?
I think you have one too many in there Aside from that, the real problem here is that these old codegen tools make some deeeeeeep assumptions about the ability to string-join various arguments and get the right results. It's awful. I was trying to make it simpler in this script, which I did for all of the in-tree consumers, but out of tree people apparently don't always follow the same conventions. For example, we generate clients for every directory that requests it. For a reason I do not know, you have files like which say but you don't actually want a client? If you specify then it t
So this code expects that you generate an omnibus client. What if I add a flag, like which you can use to call it for each API to make distinct clients? In truth the codegen tools need to be fixed, but that has SO MUCH ripple I just can't right now.
Code change:
# Deepcopy # local input_pkgs=() <del> while read -r file; do dir="$(dirname "${file}")" </del> <ins> while read -r dir; do </ins> pkg="$(cd "${dir}" && GO111MODULE=on go list -find .)" input_pkgs+=("${pkg}") done < <( <del> ( kube::codegen::internal::git_grep -l  </del> <ins> ( kube::codegen::internal::git_grep -l --null  </ins> -e '+k8s:deepcopy-gen='  ":(glob)${root}"/'**/*.go'  || true  <del> ) | LC_ALL=C sort -u </del> <ins> ) | xargs -0 -n1 dirname  | LC_ALL=C sort -u </ins> ) if [ "${#input_pkgs[@]}" != 0 ]; then

Related discussion:
text: As the codegen was breaking us at several step, hence didn't prepared the PR.
Thanks for the PR. I have one more concern, is it possible to call and client-gen on the same pkg? With the current script, that is not possible because of the line It removes all the generated files first and then calls the . Hence the later clientset gen will remove the earlier generated clients.
Why are you generating a client for an internal version that isn't served?
My mistake. We need to stop using clients for internal versions.
Code change:
# See the License for the specific language governing permissions and # limitations under the License. <del> FROM alpine:latest </del> <ins> FROM BASEIMAGE CROSS_BUILD_COPY qemu-QEMUARCH-static /usr/bin/ </ins> COPY nnp /usr/local/bin/nnp RUN chmod +s /usr/local/bin/nnp

Related discussion:
text: <!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature This issue is for creating image as multiarch
/assign
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig testing
Code change:
// true. SyncKnownPods(desiredPods []*v1.Pod) map[types.UID]PodWorkType <ins> // IsPodKnownTerminated returns true if the provided pod UID is known by the pod // worker to be terminated. If the pod has been force deleted and the pod worker // has completed termination this method will return false, so this method should // only be used to filter out pods from the desired set such as in admission. // // Intended for use by the kubelet config loops, but not subsystems, which should // use ShouldPod*(). IsPodKnownTerminated(uid types.UID) bool </ins> // CouldHaveRunningContainers returns true before the pod workers have synced, // once the pod workers see the pod (syncPod could be called), and returns false // after the pod has been terminated (running containers guaranteed stopped).

Related discussion:
text: There appears to be a change of behavior / possible regression in k8s 1.22 with respect to pod lifecycle behavior. We've started to see some cases when multiple pods are being created (often as part of a k8s job) the following will happen: pods get created by e.g. job controller (e.g. say 5). Each pod in job requests majority of node memory so only one such pod can fit on node at a time. schedules [first] pod to node, rest (4 pods) stay in pending accepts first pod, runs it to completion pod get scheduled to the node rejects the pod during pod admission, e.g. with , despite the fact that the first pod has already completed successfully This results in many of pods backing the job getting into Failed phase and having to be recreated by job controller. The job can often get into BackofLimit exceeded to hitting backoffLimit. The root issue seems to be a mismatch of when kubelet actually considers the pod to complete and what it reports to API server (which causes the followup pods to be scheduled to it). I'm still investigating further, but this may be related to the pod lifecycle changes in 1.22 -- I've created a e2e test as minimal repro of this behavior -- (Test fails) 22 build log -- 22 kubelet log -- (Test passes) 21 build log -- 21 kubelet log -- <!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --- Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.
Code change:
// true. SyncKnownPods(desiredPods []*v1.Pod) map[types.UID]PodWorkType <ins> // IsPodKnownTerminated returns true if the provided pod UID is known by the pod // worker to be terminated. If the pod has been force deleted and the pod worker // has completed termination this method will return false, so this method should // only be used to filter out pods from the desired set such as in admission. // // Intended for use by the kubelet config loops, but not subsystems, which should // use ShouldPod*(). IsPodKnownTerminated(uid types.UID) bool </ins> // CouldHaveRunningContainers returns true before the pod workers have synced, // once the pod workers see the pod (syncPod could be called), and returns false // after the pod has been terminated (running containers guaranteed stopped).

Related discussion:
text: </details
/sig node
/cc
Looking at the kubelet log in 1.22 () here's what I see -- First pod starts up... starts dies after ~10 seconds since the job is completed (the container simply runs a ) Pod completes, status is updated to API server Next pod comes in but is rejected with despite the earlier pod already being completed/terminated
did the kubelet pod lifecycle rework result in a reordering of kubelet resource bookeeping w.r.t. pod status? it seems reasonable for the scheduler to send another pod to a kubelet once it reports the pod complete and the use of the resources is done from an API perspective
I am also seeing this same behavior, but with "OutOfPods" rather than "OutOfMemory", when running a job with parallelism=100, completions=400 on GKE (which has a 110 pod per node limit).
this is easily reproducible on a local cluster as well start with this: then create this job: result on 1.22 is the job hangs with 2 completions and a bunch of rejections by the kubelet: on 1.21, the same job completes all pods successfully in a couple minutes with no OutOfpods errors
verified introduced the OutOfPods issue
Shame we didn’t have an e2e which caught this given how easy the reproducer is. This looks to be the opposite of the previous cpu admission bug (ignoring things that are terminating but not terminated). We are weak on admission testing in general. This potentially implies admission is looking at the wrong set of pods (admitted + terminating). Note there is already a bug in 1.21 and before where force deleted pods are incorrectly ignored from the set of allocated resources while they are terminating, but this doesn’t immediately seem to be that issue (since i doubt those jobs are being force deleted)
yeah, there was no force deletion involved in (or any deletion at all, actually... the completed pods just sat there, completed)
The set of admitted pods considered for pod count is everything in the pod manager filtered by filterOutTerminatingPods, which calls CouldHaveRunningContainers which returns true once a pod is seen by the workers (shortly after admission, and until we’ve confirmed via syncTerminatingPod that no containers are running). Whatever the problem is it’s likely in the interaction of that.
Code change:
// true. SyncKnownPods(desiredPods []*v1.Pod) map[types.UID]PodWorkType <ins> // IsPodKnownTerminated returns true if the provided pod UID is known by the pod // worker to be terminated. If the pod has been force deleted and the pod worker // has completed termination this method will return false, so this method should // only be used to filter out pods from the desired set such as in admission. // // Intended for use by the kubelet config loops, but not subsystems, which should // use ShouldPod*(). IsPodKnownTerminated(uid types.UID) bool </ins> // CouldHaveRunningContainers returns true before the pod workers have synced, // once the pod workers see the pod (syncPod could be called), and returns false // after the pod has been terminated (running containers guaranteed stopped).

Related discussion:
text: A pod the pod worker hasn’t seen yet is also filtered out (that means bursts of pods can get in, which seems the opposite of this problem, and is the bug i mentioned before).
Edit: one wrong thing spotted while looking - I'm not sure it fixes the issue you're seeing, but it's definitely wrong for admission (admission is filtering the known pod set down to remove pods that are terminated, but shouldn't filter pods that aren't admitted yet). This needs a node e2e reproducer that can run in serial (resource consumption is really hard to test from parallel, is slow in serial, and node e2e is back in a good state now) - / is that something either of you could help me with? I believe the reason the existing tests don't catch this is the job controller is creating pods fast enough that we see races in admission (multiple pods are admitted in one call to HandlePodAdditions) that are not caught in the current serial node e2e admission tests which rarely create more than one at a time.
Job pods still go through the scheduler to get assigned to a node, so even though the job controller can create pods in parallel, I'd still expect them to arrive at the node in an orderly fashion. It's still possible multiple arrive in a single HandlePodAdditions call, of course. It seems like kubelet admission is using laggy information that doesn't reflect pod completion the kubelet itself wrote to the pod API.
The bug turned out to be much simpler. Pods that are completed never go to the pod worker, so once a pod has reached terminal state it was being back to the list of active pods. The set of e2es that would have prevented this regression: node e2e that creates multiple pods at the same time and verifies that admission works correctly when HandlePodAdditions takes two pods at once node e2e that verifies that completed pods don't consume resources for admission purposes in the kubelet job controller e2e that attempts to verify some part of this reliably (ideally in a way that can be run in parallel e2e)
another related issue:
Linking here for completeness - a similar symptom might get exposed after this fix as described in The kubelet considers resources for terminating pods to be in use (they are!), but the scheduler ignores terminating pods and schedules new pods.
Code change:
// true. SyncKnownPods(desiredPods []*v1.Pod) map[types.UID]PodWorkType <ins> // IsPodKnownTerminated returns true if the provided pod UID is known by the pod // worker to be terminated. If the pod has been force deleted and the pod worker // has completed termination this method will return false, so this method should // only be used to filter out pods from the desired set such as in admission. // // Intended for use by the kubelet config loops, but not subsystems, which should // use ShouldPod*(). IsPodKnownTerminated(uid types.UID) bool </ins> // CouldHaveRunningContainers returns true before the pod workers have synced, // once the pod workers see the pod (syncPod could be called), and returns false // after the pod has been terminated (running containers guaranteed stopped).

Related discussion:
text: Because the kubelet now considers terminating pods, it rejects those rapidly rescheduled pods. The follow up is more involved but requires the kubelet to allow pods to be potentially admitted that can't be run currently due to those terminating pods. See the linked issue.
Code change:
NUM_ADDITIONAL_NODES=${NUM_ADDITIONAL_NODES:-} ADDITIONAL_MACHINE_TYPE=${ADDITIONAL_MACHINE_TYPE:-} <del> # Set etcd image (e.g. k8s.gcr.io/etcd) and version (e.g. v3.5.0-rc.0-0) if you need </del> <ins> # Set etcd image (e.g. k8s.gcr.io/etcd) and version (e.g. v3.5.0-0) if you need </ins> # non-default version. export ETCD_IMAGE=${TEST_ETCD_IMAGE:-} export ETCD_DOCKER_REPOSITORY=${TEST_ETCD_DOCKER_REPOSITORY:-}

Related discussion:
text: /sig scalability This issue is tracking process of upgrading etcd to 3.5.0 planned for 1.22 K8s release. Etcd has recently released Beta versions and is starting the process of correctness and performance testing. To reduce change of finding performance regression late, we will start qualifying Beta Etcd releases on GKE master branch and will upgrade to 3.5.0 as soon as it is available. Steps: [x] Do preliminary scale testing of etcd v3.5.0-beta.3 release [x] Release etcd v3.5.0-beta.3 image (, ) [x] Release etcd v3.5.0-rc.0 image () [x] Upgrade etcd server to v3.5.0-rc.0 () [x] Wait for 3.5.0 release and upgrade etcd client () [x] Release etcd v3.5.0 image (, ) [x] Upgrade etcd server to v3.5.0 /milestone v1.22
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/area etcd
Does this meant that the 3.5.0 should be promoted like
Yes, created
/area release-eng release-team /sig release For tracking: cc:
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /area release-eng release-team /sig release cc: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I do think this can be pointed in the Major Themes for the 1.22 release. I would like your opinions on this. If you do think it has a place in it, I will add it to the .
How can I be a contributor to this issue? what are the technical checklist that needs to be achieved to close out on the issue?