jiebi/CodeConvo huggingface dataset:
is recording the time that a Pod is gated. We use the timestamp when the scheduler inserts the pod into the queue: The period of time when a Pod fails PreEnque (like being gated) shouldn't be accounted in the podschedulingdurationseconds. Create a Pod with scheduling gates. Wait some time before removing the gate. Observe the metric No response <details</details<detailsAny </details<details</details<details</details<details</details<details</details
/sig scheduling
/help /triage accepted
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help /triage accepted Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
would you want to take a stab on this issue?
Yup, I can give it a shot. /assign helayoty
, as we discussed, we can address this issue with different solutions: the pod timestamp to the current time , so the gated time is substituted. timestamp . Use a pointer to the InitialAttemptTimestamp and interpret as the newest timestamp so the pod will go to the back of the queue. I'll go ahead and implement solution no. 3, wdyt?
sgtm. Any concerns ?
I think it should be option 1 + 3, and pInfo.Timestamp should be pInfo.InitialAttemptTimestamp, right? as records the time based on :
I think we should revisit this issue, in terms of its assumption: So is it reasonable to always exclude time spent in PreEnqueue from ? Note that PreEnqueue != SchedulingGates, is one of the PreEnqueue's usecases.
In other words, some user of PreEnqueue may want to include the time spent in PreEnqueue.
is a signal that cluster providers can use to build SLOs. The time spent before passing PreEnqueue is not under control of kube-scheduler, so considering it would not allow to build an SLO. Are you aware of other users of PreEnqueue?
True, but comparing to state that metric X can or cannot be used to build SLO (which is hard IMO), I'm in favor of giving suggestion on SLI/SLO's semantic definition, for example: the official SLI definition of has a footnote clearly explaining its definition: So if a pod is gated, it's not "immediately schedulable", and hence doesn't fall into the definition of podstartup_latency.
A viable solution might be providing an extra label like to explicitly exclude the time spent on preenqueue.
That is true, but there is no way today to exclude those pods from the metric, or separate the time. Maybe we need an extra metric, but I think the new metric could be the one that includes the time guarded in PreEnqueue. It should probably be part of the KEP. The old metric is already stable, so we can't add labels. And it might already be in use for SLOs.
cc
I think we should be working on that. And treat it as part of GA requirement for PodSchedulingReadiness.
An edge(maybe not edge) case is a pod's scheduling attempts is 0 but the is a long time, so prefer not to account for the waiting time when gated. For users not using PreEnqueue, they will not be impacted, but for users using PreEnqueue, it's a break change for it increases the duration time. And we can add a new metric like to measure how long a pod stays in a scheduler.
I thought of "attempts", it's just I'm not sure it's a good idea to rely on "attempts" to define a SLI/SLO, b/c "attempts" is obtained from the in-memory object and in a HA config env, it's only updated by the primary scheduler instance. In other words, the value is inaccurate when the lease owner switches.
attempt=0 is good enough for most scenarios
do we have a conclusion?
Should we move on with the solution or remove this issue and the linked PR from the current release?
I would like to move forward with this proposal. ?
To move it forward, I think we should add new metric(s) to explicitly tell it's including or excluding the time spent on PreEnqueue. Given is in stable level: We will need a new metric like "podschedulingexcludingpreenqueuedurationseconds".
28 Bug Triage Shadow here. Just checking-in, if this is still on track for the 1.28 release?
We don't have a conclusion yet, but as a bug, it should qualify for changes after the freezes.
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: use an existing azure file share name should be possible in azure file plugin What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig azure /assign
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --I noticed that the the code in for calculating the dR/dt, the derivative of R(t) with respect to t, called in the code , was not updated when the generalization for width was introduced. The should be between C and the sum of seats not the number of requests. See , where it says Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
yes, i noticed it, I am going to open a PR. /assign
/triage accepted
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: Currently in gce/config-, function get-alias-range-size returns a range which is strictly bigger than log2(max pods 2). Where equal should be also acceptable. Say if max pods constraint = 8, it should return /28, instead of /27. What you expected to happen: Return a range which is larger OR EQUAL to the max pods constraint. How to reproduce it (as minimally and precisely as possible): Take 8 as $1 param for the function. Anything else we need to know?: Need to cherrypick to past versions. Me myself will fix it.
/sig network
/cc
Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: The deployment controller for a deployment with a single replica failed to create a new ReplicaSet when the old ReplicaSet included an Evicted pod. Typically, the eviction was caused by "node low on resource: nodefs". When we manually deleted the evicted pod, the new RS and accompanying pod were created immediately. We have seen this happen at least a dozen times. Here are the audit logs for a recent case: ! What you expected to happen: The deployment controller ignores the evicted pod and creates the updated replicaset/pod. How to reproduce it (as minimally and precisely as possible): To reproduce, you would need a way to trigger a kubelet-driven eviction on a node that is currently running a single-replica Deployment with . Wait for the pod to be evicted and the new pod to be created, confirm that the evicted pod still exists, and then make a trivial change to the deployment spec. The running pod will be deleted, and no new replicaset/pod will be created. This issue does not seem to occur for manual evictions (via ); pods I evict that way disappear from the API server immediately. Anything else we need to know?: Possibly related to and/or Environment: Kubernetes version (use ): v1.8.5-gke.0 Cloud provider or hardware configuration: GKE OS (e.g. from /etc/os-release): Container-Optimized OS
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Thanks for the detailed bug report!
From Here's what happened: Deployment was updated by the user, so it scaled all old ReplicaSets to 0 old ReplicaSet was scaled to 0 (ReplicaSet's == 0), but still owned the evicted pod, which wouldn't be killed automatically because of the above comment (ReplicaSet's 0) Deployment found that there were still old Pods, and kept waiting for them to be gone ~(wait for old ReplicaSet's to be 0)~ (Edit: Deployment also checks those old Pods by listing them to make sure there are none, not just by checking ReplicaSet's status) before starting the new rollout. We can either modify Deployment controller (to ignore evicted old pods) or ReplicaSet controller (to kill evicted pods ~when it gets scaled down~), if this is not the intended behavior. WDYT?
I am actually debugging likely the same production issue with Recreate strategy in two clusters right now. (We run on patched 1.9.1.) I have traced it down to broken RS as well. in 1 cluster RS leaves around pod failed matching the selector in the other one those are evicted pods. all replicas are scaled down to 0 and report 0 in status but pods with valid controllerRefs to them are still present. Same is 0 for me but pod is still there I think the correct behaviour is for replicaSet scaled to 0 to delete all of its pod, not to leave them laying around
xref - this should fix it
master-informing: periodic-kubernetes-e2e-kind-kms ! No response /sig cluster-lifecycle
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/milestone v1.29
/assign but this test job is maintained by sig auth: /sig auth looks like the API server has decided to consistently fail on an unpolled CRB create API call from the kubeadm client. this did not happen on the kind presubmit jobs here: or kubeadm periodic jobs using the kinder tool: or my local tests. so what is different here and why are we getting informer errors? is this highlighting a bug in core k8s? cc best we can do in kubeadm is make that API call polled. FTR, the API call exactly prior to this failing API call is already polled.
Hello. I want to echo that the code freeze is starting (next week), and we want to ensure that each PR has a chance to be merged and issues can be addressed. Is this PR still targeting the 1.29 release?
FTR, the API call exactly prior to this failing API call is already polled. +1 to have the API call polled just as the API call with .
here is the kubeadm PR.
we are tracking a failing test in this ticket and we can proceed with fixing this after code freeze is in effect. i just submitted an associated fix PR here:
As the Alpha 3 release cut is approaching (2nd Nov) /priority critical-urgent
/assign
we have a couple of kubeadm passes here: of the presubmit variant of this job: pull-kubernetes-e2e-kind-kms this seems like a flake and not a failure, given the offending PR merged on the 27th and looking at the history of the periodic:
we are having the similar issue with i saw in the logs I am not sure what I should do, could you help me?
thanks
Can you pls tell where is the the log you pasted above? I searched in this file: , and some other logs related with kubeadm but cannot find it. The original issue is expected to be fixed by: It's now continuous green now: But the log you pasted do point to the change we landed last week:
kubeadm introduced a new step in the "init" process. it creates an file but it needs RBAC for it. this RBAC is created in a rather complicated manner (with a fallback), but in your case it's failing at the first step where the API server is not reachable in 40 seconds. do you expect the API server to no be ready in 40 seconds? are you sure there are no other problems with the setup and it just ends up failing at this first API server step where RBAC is created? other jobs like CAPI (non-upgrade), kinder, kind are green with respect to this. i wonder if we can just stop returning this error and fallback to the super- "create" attempt later: this will allow for more time for certain setups to start - 40 + 40 seconds. ... it exactly to catch problems like the one is reporting - no early connectivity to the API server. but if this is now the "first thing" in kubeadm, it might have introduced a bit of a timeout regression for slower setups.
also note that the kubeadm CP provider in CAPI does not yet support this feature and the upgrade CAPI jobs are also failing:
actually, the feature seems change the order of "init": it waits for the apiserver + kubelet (, 40seconds for kubelet) and the RBAC create ( when the first client() creation happens) at the same time. i think what has to be changed here is: the parallel aspect of the apiserver + kubelet wait. it has been difficult to manage and confusing. delay the RBAC create to be after the apiserver + kubelet waiting is done. maybe something we can do for .29 is 2. but 1 is a bigger refactor.
PR with an attempt to improve the kubeadm wait logic: i do think CAPG has unrelated problems. the kubelet might be failing to start or starting but not creating the static pod for kube-apiserver.
I stumbled across this while experimenting with kubeconfig load order/merging. There are various helpers constructed from a Factory that use , however this always uses the same default ClientConfig because doesn't use the passed in version param: Am I missing something or is this a bug? /cc
Looks like a bug, clientforversion should accept "" (which can provide a default) and SetKubernetesDefaults has the final fallback protection (by using latest.Version). If version != defaultConfig.Version, we should copy defaultConfig, set version, then apply kubedefaults. The reason we take version here is if a resource is specified via file which only exists in a newer version, ie user says: --api-version=v1beta1 -f RESTMapper can handle that newresource is only valid for v1beta3 and apply the right override, then ask for client. --api-version is expected to bubble up via default config.
/priority failing-test /priority critical-urgent /sig apps This test case has been failing for a while and affects a number of jobs: This is affecting multiple jobs on the testgrid dashboard, and prevents us from cutting 1.9.0-alpha.2 (kubernetes/sig-release). Is there work ongoing to bring this job back to green?
All test failures look similar: . ~Is it due to network issue instead of DaemonSet?~
This seems related to the new DaemonSet. The connection refused error is happening because kube-apiserver is crashing:
needs an update
Ah, well... I guess it's halfway working as intended. We put that there to make the tests fail if we forgot to update it when adding a new version. We had hoped it would fail during presubmit tests though. :(
Sending a fix now
I wonder why the test is not failing on submit queue tests. Do i need to fix deployment and replicaset?
The rest of the group is not merged, and the v1 PR takes the selector mutability into account.
Since this bug crashes apiserver I'd expect it to affect many other e2e tests.
Noted, we need to fix deployment and replicaset when the v1 PR is merged.
Not quite sure how this test isn't failing before but is now consistently failing. I'd expect the commit that triggers this to block the submit queue. The e2e test is still accessing DaemonSet through extensions/v1beta1 endpoints. From the above apiserver log it's hard to tell what updates DaemonSet via apps/v1 before the code panics. I tried to reproduce it locally, and found that garbage collector PUT apps/v1 DaemonSet:
When apps/v1 gets introduced, it may be ordered before apps/beta1 alphabetically. IIRC, there is a code change on the apps/v1 PR that changes the order of API versions such that apps/v1 is ranked before apps/v1beta2 and apps/v1beta2 before apps/v1beta1. Perhaps this problem can be fixed when the PR merges.
Since crashes apiserver and makes debugging harder, which defeats the purpose of using it, we should return error on unsupported versions instead.
The first run with this change is still in progress, but this test already passed: https://k8s- Thanks for the quick fix
Kubectl client Kubectl api-versions should check if v1 is in available api versions [Conformance] Expected error: <_errors.errorString | 0xc208641770: { s: "Error running &{/jenkins-master-data/jobs/kubernetes-upgrade-gke-1.1-master-step1-deploy/workspace/kubernetes/platforms/linux/amd64/kubectl [kubectl --server=https://130.211.136.232 --kubeconfig=/var/lib/jenkins/jobs/kubernetes-upgrade-gke-1.1-master-step1- api-versions] [] error: Couldn't get available api versions from server: no kind "APIVersions" is registered for version "v1"n [] 0xc208382fa0 exit status 1 true [0xc208627148 0xc208627168 0xc208627188] [0xc208627148 0xc208627168 0xc208627188] [0xc208627160 0xc208627180] [0x6d81a0 0x6d81a0] 0xc2082fb2c0}:nCommand stdout:nnstderr:nerror: Couldn't get available api versions from server: no kind "APIVersions" is registered for version "v1"nn", } Error running &{/jenkins-master-data/jobs/kubernetes-upgrade-gke-1.1-master-step1-deploy/workspace/kubernetes/platforms/linux/amd64/kubectl [kubectl --server=https://130.211.136.232 --kubeconfig=/var/lib/jenkins/jobs/kubernetes-upgrade-gke-1.
1-master-step1- api-versions] [] error: Couldn't get available api versions from server: no kind "APIVersions" is registered for version "v1" [] 0xc208382fa0 exit status 1 true [0xc208627148 0xc208627168 0xc208627188] [0xc208627148 0xc208627168 0xc208627188] [0xc208627160 0xc208627180] [0x6d81a0 0x6d81a0] 0xc2082fb2c0}: Command stdout: stderr: error: Couldn't get available api versions from server: no kind "APIVersions" is registered for version "v1" not to have occurred* See and Investigating. cc
Okiedokie. Looks like the v1.1 client is getting a response from the v1.2 master, but the v1.1 client doesn't know how to handle the v1 APIVersions object, because in v1.1, that object was never expected to have an . From a 1.1 server, we get which doesn't include a . We should expect to have plenty of pre-v1.1.7 clients floating around, and releasing a fixed v1.1.8 probably isn't going to solve this; if we want to be backward compatible, we need to not return an string with unversioned kinds like . Looking into more details about how we got here... cc cc
I have a maybe feeling that this is caused by ? In particular, we went from: to
Probably it's caused by this:
Let me double check the code. cc
What about that line? I've never dealt with the version negotiation; should the server and client be negotiating in a way that the server can return an object without ?
More precisely it's this line: This line sets the version to "v1". In , during encoding, we set unversioned object's APIVersion field to "v1".
I think in the long run, because we expect to move many unversioned types to group "common", so they should be versioned, so making APIVersion=v1 is the right move. Back to the backwards compatibility issue, how's the situation at other url, like http://127.0.0.1:8080/apis/extensions?
That address also returns a versioned object, which at release 1.1 is unversioned: Do we need to keep all unversioned objects unversioned, or do we only need to ensure the server returns unversioned object at api/?
cc
Per our versioning rules, we should expect a v1.2 server to remain backward compatible with v1.1 clients. I don't have much experience with the api machinery, but if clients weren't expecting an unversioned object, it seems like we can't up and switch to versioned objects.
I think we have to maintain compatibility somehow. any ideas for a fix other than the obvious one?
Yes, we have to maintain backward compatibility with 1.1 (and 1.0) clients. New paths that old clients didn't know about could behave differently than paths that existed in previous releases.
Nope, let's make sure we leave a test and comment.
We can't make the change here though, this would break "versioned unversioned" objects (like )
The other implication of course is now we "versioned unversioned" objects (v1 status) and "unversioned unversioned", which means we shouldn't ever add new objects that don't return a version. I don't think anything in 1.2 qualifies as a new object without a version, just 1.1 objects that didn't have correct versions. Having objects from extensions group be returned without versions terrifies me - I don't see any way of describing that except as a bug.
We may need per API group version rules registered on scheme about whether "unversioned" objects are returned in their scheme version or as empty. That's probably the backward compatible fix that gives us a hope of reasoning about behavior correctly (status in 1.0 is v1, status in extensions might be "", apiversions in any new group is "v1"). The unversioned marker on the objects should ensure 1.2 clients can receive objects in both "" and "v1" form, we should add a test to verify that. I'm against letting empty API versions spread, but agree backcompat for existing groups versions should match previous behavior.
Can we make (or use) a different serializer/codec just for the endpoints that 1.1 clients used?
Probably we just need to change the version registered here:
That won't work like you expect - the type registry then has no knowledge in v1 of what schema the object is. That's not what we want - we want to not serialize the version of the object for specific API groups, which is a much more targeted change. This is a special, exceptional case (v1 group should return v1 Status. v1beta1 should not). , Chao Xu wrote:
Changing v1 breaks 1.0 backwards compat (we always returned apiVersion=v1 for 1.0) , Clayton Coleman wrote:
I mean this: Will this work?
I'm not sure without looking at the outcome (which is why unversioned is a bit of a disaster in our codebase). I would much prefer to explicitly identify which objects should have an unset apiversion via a method on the scheme, and do the explicit check at the time we do convert to version. That's an extremely controlled change, is obvious, and doesn't make the unversioned problem worse.
I remember somewhere in asserts version cannot be empty, so I guess my proposed solution won't work without extra work. I'll take the approach suggested by