jiebi/CodeConvo huggingface dataset:
Code change:
} return nil, framework.AsStatus(err) } <ins> // Check if the returned victims are valid. for nodeName, victims := range nodeNameToVictims { if victims == nil || len(victims.Pods) == 0 { if extender.IsIgnorable() { delete(nodeNameToVictims, nodeName) klog.InfoS("Ignoring node without victims", "node", nodeName) continue } return nil, framework.AsStatus(fmt.Errorf("expected at least one victim pod on node %q", nodeName)) } } </ins> // Replace victimsMap with new result after preemption. So the // rest of extenders can continue use it as parameter. victimsMap = nodeNameToVictims

Related discussion:
text: If yes, then one hypothesis is that those custom filters filter the node in the filter phase, but not when executed in the preemption phase, and so this will result in a candidate node with no victims?
Yes, it includes filter plugins and uses cycle state.
you need to make sure that those filter plugins play nicely when executed again in the preemption phase in the same cycle: i.e., produce the same result.
Thanks What do you mean by performing filter in the preemption phase? would you mind elaborating it a little?
What additional logic is needed to handle the preemption case?
in the preemption phase we run the filters again to check if removing lower priority pods make the node schedulable: we later add them one by one to reduce the set of victim pods to the absolute minimum: My hypothesis is that somehow your filter returns success when adding all the pods back, and so you end up with zero victims and a candidate node. This shouldn't happen because if the pod fits the node without removing any victims, then we shouldn't be running preemption in the first place. So the theory is that the custom filter is returning false when run in the filter phase, but returns true when executed in the preemption phase perhaps because it makes some assumptions about cyclestate that makes it behave this way (e.g. that a the filter will be executed once per node in a scheduling cycle).
Disabled the filter plugin, but still see the same issues.
are you able to change the scheduler code and run the test again? I can send a patch tomorrow to add some debugging messages to help us root cause the issue.
yes, thanks a lot! really appreciate it!
You are right. We still use a deprecated Predicate extender, which causes the problem. After disabling it, it's working fine. It used to work with 1.18 though. We are retiring it. Thank you so much!!!
If the test cases can help debug scheduler extender Webhook (Predicate) too, I'd like to try it. Thanks again!
How can we improve the preemption logic to prevent bugs and issues in custom Filters or Predicates crashing the scheduler?
for starter, we should check the list len, the scheduler should not panic in all cases.
Code change:
} return nil, framework.AsStatus(err) } <ins> // Check if the returned victims are valid. for nodeName, victims := range nodeNameToVictims { if victims == nil || len(victims.Pods) == 0 { if extender.IsIgnorable() { delete(nodeNameToVictims, nodeName) klog.InfoS("Ignoring node without victims", "node", nodeName) continue } return nil, framework.AsStatus(fmt.Errorf("expected at least one victim pod on node %q", nodeName)) } } </ins> // Replace victimsMap with new result after preemption. So the // rest of extenders can continue use it as parameter. victimsMap = nodeNameToVictims

Related discussion:
text: Other than that, I think we need to clearly document what the preemption logic does (that it executes the prefilter extensions points and the filter plugins multiple times) and the expectations from filter plugins (that they may get executed more than once in the same cycle)
Is it a predicate extender or preemption extender? If it's a predicate extender, I don't quite think we found the root cause - b/c predicate extenders are not invoked during preemption (). It makes more sense if it's a preemption extender - if the preemption extender mutates the to make it invalid. If it's the case, could you verify it by adding debug info before and after L157 to check the validness of ?
That's always true :)
filed a PR to mitigate it.
It's a predicate extender. I'll debug it more.
Added some debug info. after and . Here's the result. returned 4 candidates, but one of candidates () has zero pod. The candidate happened to be the best candidate and hence caused the "" error. It should have nothing to do with since our extender doesn't support Preemption. Disabling it may change pod placements/candidates selection and bypassed the error. It's like a coincidence. To remove the panic, a simple fix is to check if is empty. If it is, do not add the node to candidates. Alternatively, a change to may work too. Here is the log with debug info.
This looks more promising. Back to digging into the root cause, it seems returns an empty but with . So that implies firstly we have a non-nil , and then in , the is separated into two list: violatingVictims and nonViolatingVictims. Finally the two victims list are tried one by one to run . It looks like returns {fits=true, nil error} for every victim, which is quite abnormal. I think it's problematic that every pod is retrievable and in which we can debug into.
Yes, should not return fit=true with empty pods in the first place, but it's hard to locate the problematic code in this function.
The following code in looks questionable to me. If is empty, should the function return nil, nil? Also, if is true for all extenders, the loop won't do anything and candidates will remain the same.
Code change:
} return nil, framework.AsStatus(err) } <ins> // Check if the returned victims are valid. for nodeName, victims := range nodeNameToVictims { if victims == nil || len(victims.Pods) == 0 { if extender.IsIgnorable() { delete(nodeNameToVictims, nodeName) klog.InfoS("Ignoring node without victims", "node", nodeName) continue } return nil, framework.AsStatus(fmt.Errorf("expected at least one victim pod on node %q", nodeName)) } } </ins> // Replace victimsMap with new result after preemption. So the // rest of extenders can continue use it as parameter. victimsMap = nodeNameToVictims

Related discussion:
text: Why do we bother creating a new candidates later? Can it just return candidates?
It can prevent from returning candidates with empty pods, but what about ? It's possible to add nodes without victim pods to candidates if returned by contain such nodes. How about adding the following check to CallExtender?
Updated the PR based on the finding and discussions.
Yes, we can use a flag to mark if all extenders don't support preemption. If yes, simply return the candidates.
Technically this can exist due to faulty extender implementation. If we really want to guard it, instead of logging the error and continue, I'm more inclined to return the error immediately as this is a fatal error - the victimsMap cannot be used either for latter extender or further preemptor nominating.
Is my understanding correct? In , if victimsMap is empty, should it return instead of ?
The only case victimsMap can be empty is empty candidates, and so return either doesn't quite matter, right?
You are right.
As long as one extender returns an invalid victimsMap (with empty pods), s returns an error immediately? What about ? Should it return an error immediately if it returnswith empty pods?
I think so. Because the result it returned is faulty, and we don't want to continue based on it The same. We may don't want to continue the scheduling cycle based on faulty result.
Code change:
}, }, }, <del> "spec.timeZone: Invalid value: "AMERICA/new_york": unknown time zone AMERICA/new_york": { ObjectMeta: metav1.ObjectMeta{ Name:      "mycronjob", Namespace: metav1.NamespaceDefault, UID:       types.UID("1a2b3c"), }, Spec: batch.CronJobSpec{ Schedule:          "0 * * * *", TimeZone:          &timeZoneBadCasing, ConcurrencyPolicy: batch.AllowConcurrent, JobTemplate: batch.JobTemplateSpec{ Spec: batch.JobSpec{ Template: validPodTemplateSpec, }, }, }, }, </del> "spec.timeZone: Invalid value: " America/New_York": unknown time zone  America/New_York": { ObjectMeta: metav1.ObjectMeta{ Name:      "mycronjob",

Related discussion:
text: test withnot FAIL In a macos. do or golang upstrem disscusion. We use time.LoadLocation to validate timezone setting in object spec like cronjob.(It is ok) We also add a test try to validate is not a crroct value. This test is not true sometimes, for example, in default macos this is a true time_zone value.(maybe also true in other case insensitive os, but I have no device to try) At least we should get the test to pass on macos, I hope. <details</details<details</details<details</details<details</details<details</details<details</details
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /sig test (I guess) Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig testing (I guess)
see
/remove-sig testing /sig apps
I was very surprised to find that this error only occurs on darwin, and does not occur on windows, although windows is also case-sensitive by default. When using , windows finally use (see ), but darwin and linux have more choices (see ) they will first try , using the method, which will call the system's open method to open , because darwin is case-insensitive by default, and this operation will succeed.
/triage accepted
Code change:
}() } <del> // set up stdout stream </del> if e.stdout != nil { <del> headers.Set(api.StreamType, api.StreamTypeStdout) remoteStdout, err := conn.CreateStream(headers) if err != nil { return err } </del> wg.Add(1) go func() { defer wg.Done()

Related discussion:
text: We create the stdin stream and immediately start copying local stdin to the container. We do the same for stdout and stderr (create stream, start copying immediately). In the kubelet, we wait to receive all the streams before we start the exec and copy between the streams and the container. The spdy connection frame handling code has 5 frame workers, each with a queue of up to 50 frames. In the event that we try to push more than 50 frames to a single queue before all streams are ready, the 51st attempt to push a frame blocks until there is space available in the queue. But there won't be any space until the server starts reading from the streams, which won't happen until all the streams have been received, so we're stuck at this point. The fix is to create all the streams first, and only start copying after all the streams have been created and the server is ready to copy between the streams and the container. Originally reported here:
FYI
This seems pretty high priority.
It's an easy fix. has already tested the patch for OpenShift and it works. Are we willing to pull it in before is resolved, as that will take longer to put in place?
Here's the patch, for reference:
5 * 50 = 250 frames before you get stuck? Or did I misunderstand?
Oh, I guess all the frames go to the same worker for some reason?
anyway, I'm fine with taking this and cherry picking it. Order doesn't matter; someone has to rebase either way, right?
frames are assigned to each queue based on stream id, so if you sent 50 frames to the stdin stream, that queue fills up and the loop that processes incoming frames is blocked until something starts reading from the stream. Working on a PR now.
Gotcha, thanks. , Andy Goldstein wrote:
an update: I spent some time trying to create an e2e that would fail more often than not without this fix, but I haven't had any luck so far (I have it stream a copy of the kubectl binary from the local system to the pod). The issue that originally uncovered the problem was from running exec on a Windows system, where it reproduces consistently, but I haven't seen it fail much on Linux. I will keep trying...
Code change:
<polygon style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" points="975.431,762.352 965.093,758.095 970.506,756.696 972.634,751.527 "/> </g> <g> <del> <path style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" d="M 478.975 491 C 552.046,491 637.129,312.5 711.464,312.5"/> <polygon style="fill: #000000" points="478.975,486 468.975,491 478.975,496 "/> <polygon style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" points="478.975,486 468.975,491 478.975,496 "/> </del> <ins> <path style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" d="M 478.974 491 C 552.046,491 637.13,312.5 711.464,312.5"/> <polygon style="fill: #000000" points="478.974,486 468.974,491 478.974,496 "/> <polygon style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" points="478.974,486 468.974,491 478.974,496 "/> </ins> <polygon style="fill: #000000" points="718.964,312.5 708.964,317.5 711.464,312.5 708.964,307.5 "/> <polygon style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" points="718.964,312.5 708.964,317.5 711.464,312.5 708.964,307.5 "/> </g> <g> <del> <path style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" d="M 478.933 505.457 C 677,522.75 650,676.75 712.305,704.479"/> <polygon style="fill: #000000" points="479.368,500.476 468.971,504.587 478.498,510.438 "/> <polygon style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" points="479.368,500.476 468.971,504.587 478.498,510.438 "/> </del> <ins> <path style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" d="M 478.932 505.457 C 677,522.75 650,676.75 712.305,704.479"/> <polygon style="fill: #000000" points="479.367,500.476 468.97,504.587 478.497,510.438 "/> <polygon style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" points="479.367,500.476 468.97,504.587 478.497,510.438 "/> </ins> <polygon style="fill: #000000" points="719.157,707.529 707.988,708.031 712.305,704.479 712.054,698.895 "/> <polygon style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" points="719.157,707.529 707.988,708.031 712.305,704.479 712.054,698.895 "/> </g>

Related discussion:
text: Architecture picture: says kubecfg is deprecated. The architecture picture should be updated to show kubectl instead of kubecfg.
ref /cc
Code change:
} // NewPodInformer creates a shared index informer that returns only non-terminal pods. <del> func NewPodInformer(client clientset.Interface, resyncPeriod time.Duration, schedulerName string) coreinformers.PodInformer { </del> <ins> func NewPodInformer(client clientset.Interface, resyncPeriod time.Duration) coreinformers.PodInformer { </ins> selector := fields.ParseSelectorOrDie( <del> "spec.schedulerName=" + schedulerName + ",status.phase!=" + string(v1.PodSucceeded) + </del> <ins> "status.phase!=" + string(v1.PodSucceeded) + </ins> ",status.phase!=" + string(v1.PodFailed)) lw := cache.NewListWatchFromClient(client.CoreV1().RESTClient(), string(v1.ResourcePods), metav1.NamespaceAll, selector) return &podInformer{

Related discussion:
text: <!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Scheduler's pod informer has a selector that . This is incorrect. Scheduler must watch all the pods of the cluster, including those scheduled by other schedulers in order to update its cache of the state of the cluster. Otherwise, it will miss important events and the amount of available resources, pods on nodes, ... which are needed for scheduler to work correctly, will not be updated properly. /sig scheduling /assign
cc
For updating it cache, shouldn't the default scheduler rely on node info/status? Also its not very clear, what information the default scheduler needs from other pods (not to be scheduled by it) that it makes decision on? Just trying to understand the issue more clearly.
Scheduler watches informer events to update its cache. Node status is not enough to update all the information that scheduler caches. Scheduler must know what pods are placed on what nodes. For example, in order to check inter-pod affinity/anti-affinity it needs to know the placement of all the pods, including those scheduled by other schedulers.
But this information is present here: So it seems that your PR will now make sure to get the scheduled pods' information directly from the pod informer.
So when I check the history, I see it was here:
have we seen any specific issues due to this change (like between default scheduler and DS controller)?
Yes, but scheduler relies on its pod informers to keep this location up to date. That's exactly why it is important for the scheduler to receive updates for all the pods in the cluster. Yes. We missed the problem at the time. This is problematic only in clusters that use multiple schedulers. I initially thought that DS controller could cause issues as well, but DS controller does not set , so it must not cause an issue. We have seen instances of scheduler's missing certain events in cluster and as a result scheduler's cache getting stale, but I doubt if it is related to this particular issue. This only affects clusters with more than one scheduler. If you run the test that I have written in with the head, it will fail. That tests simulate a multi-scheduler scenario.
Code change:
return $KubeEnv.Contains('EXEC_AUTH_PLUGIN_URL') } <ins> # Permanently adds a directory to the $env:PATH environment variable. function Add-MachineEnvironmentPath { param ( [parameter(Mandatory=$true)] [string]$Path ) # Verify that the $Path is not already in the $env:Path variable. $pathForCompare = $Path.TrimEnd('').ToLower() foreach ($p in $env:Path.Split(";")) { if ($p.TrimEnd('').ToLower() -eq $pathForCompare) { return } } $newMachinePath = $Path + ";" + ` [System.Environment]::GetEnvironmentVariable("Path","Machine") [Environment]::SetEnvironmentVariable("Path", $newMachinePath, ` [System.EnvironmentVariableTarget]::Machine) $env:Path = $Path + ";" + $env:Path } </ins> # Export all public functions: Export-ModuleMember -Function *-*

Related discussion:
text: <!-- Please only use this template for submitting enhancement requests --What would you like to be : pigz for GCE Windows Why is this needed: Pigz improves the image layer extraction performance for multi-core systems. Pigz is usually preinstalled on Linux. This enhancement is scoped specifically for the GKE Windows node scripts. For Context:
/assign me
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
GitHub didn't allow me to assign the following users: me. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign me Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig windows /assign jeremyje
s/GKE/GCE/? as in kube-up?
Replaced but I don't understand the kube-up part.
is how one creates a GCE cluster from this repo. Otherwise we'd be talking about something that isn't part of kubernetes.
Code change:
etcd_servers: $MASTER_IP roles: - kubernetes-pool <ins> - kubernetes-pool-vagrant </ins> cbr-cidr: $MINION_IP_RANGE minion_ip: $MINION_IP EOF

Related discussion:
text: As a sysadmin When using the Vagrant cluster provided And logged in as the vagrant user to the master or minions I would like to be able to call docker commands wihout requiring the 'sudo' prefix. For Fedora =19 and RHEL=7 this can be enabled by adding the vagrant user to the docker group in /etc/groups
This should be trivial, but its tricker to do because of the whole salt configuration process as the docker group is only created on each kubernetes-minion once docker is installed by the salt-minion. To do this properly, we would need a salt state change for this, and new flag in to let you specify additional users that should be in the docker group.
Code change:
cpuQuery                  = "Processor(_Total)% Processor Time" memoryPrivWorkingSetQuery = "Process(_Total)Working Set - Private" memoryCommittedBytesQuery = "MemoryCommitted Bytes" <del> // Perf counters are updated every second. This is the same as the default cadvisor collection period // see https://github.com/google/cadvisor/blob/master/docs/runtime_options.md#housekeeping perfCounterUpdatePeriod = 1 * time.Second </del> <ins> // Perf counters are updated 10 seconds. This is the same as the default cadvisor housekeeping interval // set at https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/cadvisor/cadvisor_linux.go perfCounterUpdatePeriod = 10 * time.Second </ins> // defaultCachePeriod is the default cache period for each cpuUsage. // This matches with the cadvisor setting and the time interval we use for containers. // see https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/cadvisor/cadvisor_linux.go#L63

Related discussion:
text: Perf counters are currently updated every second which doesn't match the default cadvisor collection period as 10 seconds. Increase the update interval of perf counters to 10 seconds to match the house keeping interval of cadvisor. The fresh install of our internal AKS clusters. The perf counter query (pdh!PdhCollectQueryData) is responsible for 20% of CPU time in kubelet process on internal clusters, which can be reduced by lowering the perf counter update interval. <details</details<detailsAzure Kubernetes Service (AKS) </details<detailsOperating System: Windows Server 2019 Datacenter OS Build Number: .3770.amd64fre.rs5release.-1434 </details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node /sig windows
/area cadvisor Since it is a performance overhead and not a bug per se, I will change the kind of this issue: /remove-kind bug /kind feature I bet there is an issue for this already. Can you comment/dedup?
I believe the ask here is to increase the which is only used on Windows to fetch stats from windows specific per counters. Windows does not use cAdvisor and instead uses which reads from the perf counters directly. /cc
(removing cadvisor since it is ask for windows stats) /remove-area cadvisor
agreed that is the ask. I don't have any major issue with it but would be worried about change in behavior for existing users. What are our options for making this configurable?
That would be great if making it configurable.
It might not be worth the overhead to pass this as configuration through kubelet parameters. I am unsure what affect it would have on folks who rely on it for current reporting or if it would be ok to bump the querying time. if you increase it to 10s do you see a large drop in cpu usage?