jiebi/CodeConvo huggingface dataset:
Code change:
offset, numCandidates := pl.getOffsetAndNumCandidates(int32(len(nodeInfos))) candidates, _ := dryRunPreemption(context.Background(), fwk, state, tt.pod, nodeInfos, nil, offset, numCandidates) s := SelectCandidate(candidates) <ins> if s == nil || len(s.Name()) == 0 { return } </ins> found := false for _, nodeName := range tt.expected { if nodeName == s.Name() {

Related discussion:
text: If yes, then one hypothesis is that those custom filters filter the node in the filter phase, but not when executed in the preemption phase, and so this will result in a candidate node with no victims?
Yes, it includes filter plugins and uses cycle state.
you need to make sure that those filter plugins play nicely when executed again in the preemption phase in the same cycle: i.e., produce the same result.
Thanks What do you mean by performing filter in the preemption phase? would you mind elaborating it a little?
What additional logic is needed to handle the preemption case?
in the preemption phase we run the filters again to check if removing lower priority pods make the node schedulable: we later add them one by one to reduce the set of victim pods to the absolute minimum: My hypothesis is that somehow your filter returns success when adding all the pods back, and so you end up with zero victims and a candidate node. This shouldn't happen because if the pod fits the node without removing any victims, then we shouldn't be running preemption in the first place. So the theory is that the custom filter is returning false when run in the filter phase, but returns true when executed in the preemption phase perhaps because it makes some assumptions about cyclestate that makes it behave this way (e.g. that a the filter will be executed once per node in a scheduling cycle).
Disabled the filter plugin, but still see the same issues.
are you able to change the scheduler code and run the test again? I can send a patch tomorrow to add some debugging messages to help us root cause the issue.
yes, thanks a lot! really appreciate it!
You are right. We still use a deprecated Predicate extender, which causes the problem. After disabling it, it's working fine. It used to work with 1.18 though. We are retiring it. Thank you so much!!!
If the test cases can help debug scheduler extender Webhook (Predicate) too, I'd like to try it. Thanks again!
How can we improve the preemption logic to prevent bugs and issues in custom Filters or Predicates crashing the scheduler?
for starter, we should check the list len, the scheduler should not panic in all cases.
Code change:
offset, numCandidates := pl.getOffsetAndNumCandidates(int32(len(nodeInfos))) candidates, _ := dryRunPreemption(context.Background(), fwk, state, tt.pod, nodeInfos, nil, offset, numCandidates) s := SelectCandidate(candidates) <ins> if s == nil || len(s.Name()) == 0 { return } </ins> found := false for _, nodeName := range tt.expected { if nodeName == s.Name() {

Related discussion:
text: Other than that, I think we need to clearly document what the preemption logic does (that it executes the prefilter extensions points and the filter plugins multiple times) and the expectations from filter plugins (that they may get executed more than once in the same cycle)
Is it a predicate extender or preemption extender? If it's a predicate extender, I don't quite think we found the root cause - b/c predicate extenders are not invoked during preemption (). It makes more sense if it's a preemption extender - if the preemption extender mutates the to make it invalid. If it's the case, could you verify it by adding debug info before and after L157 to check the validness of ?
That's always true :)
filed a PR to mitigate it.
It's a predicate extender. I'll debug it more.
Added some debug info. after and . Here's the result. returned 4 candidates, but one of candidates () has zero pod. The candidate happened to be the best candidate and hence caused the "" error. It should have nothing to do with since our extender doesn't support Preemption. Disabling it may change pod placements/candidates selection and bypassed the error. It's like a coincidence. To remove the panic, a simple fix is to check if is empty. If it is, do not add the node to candidates. Alternatively, a change to may work too. Here is the log with debug info.
This looks more promising. Back to digging into the root cause, it seems returns an empty but with . So that implies firstly we have a non-nil , and then in , the is separated into two list: violatingVictims and nonViolatingVictims. Finally the two victims list are tried one by one to run . It looks like returns {fits=true, nil error} for every victim, which is quite abnormal. I think it's problematic that every pod is retrievable and in which we can debug into.
Yes, should not return fit=true with empty pods in the first place, but it's hard to locate the problematic code in this function.
The following code in looks questionable to me. If is empty, should the function return nil, nil? Also, if is true for all extenders, the loop won't do anything and candidates will remain the same.
Code change:
offset, numCandidates := pl.getOffsetAndNumCandidates(int32(len(nodeInfos))) candidates, _ := dryRunPreemption(context.Background(), fwk, state, tt.pod, nodeInfos, nil, offset, numCandidates) s := SelectCandidate(candidates) <ins> if s == nil || len(s.Name()) == 0 { return } </ins> found := false for _, nodeName := range tt.expected { if nodeName == s.Name() {

Related discussion:
text: Why do we bother creating a new candidates later? Can it just return candidates?
It can prevent from returning candidates with empty pods, but what about ? It's possible to add nodes without victim pods to candidates if returned by contain such nodes. How about adding the following check to CallExtender?
Updated the PR based on the finding and discussions.
Yes, we can use a flag to mark if all extenders don't support preemption. If yes, simply return the candidates.
Technically this can exist due to faulty extender implementation. If we really want to guard it, instead of logging the error and continue, I'm more inclined to return the error immediately as this is a fatal error - the victimsMap cannot be used either for latter extender or further preemptor nominating.
Is my understanding correct? In , if victimsMap is empty, should it return instead of ?
The only case victimsMap can be empty is empty candidates, and so return either doesn't quite matter, right?
You are right.
As long as one extender returns an invalid victimsMap (with empty pods), s returns an error immediately? What about ? Should it return an error immediately if it returnswith empty pods?
I think so. Because the result it returned is faulty, and we don't want to continue based on it The same. We may don't want to continue the scheduling cycle based on faulty result.
Code change:
} By("Reading value under foo from member with index 2") <del> if v := pet.read(2, "foo"); v != "bar" { framework.Failf("Read unexpected value %v, expected bar under key foo", v) </del> <ins> if err := pollReadWithTimeout(pet, 2, "foo", "bar"); err != nil { framework.Failf("%v", err) </ins> } }) })

Related discussion:
text: https://k8s- Failed: [] PetSet [Slow] [Feature:PetSet] [] Deploy clustered applications [Slow] [Feature:PetSet] should creating a working redis cluster [Feature:PetSet] {Kubernetes e2e suite}
The test failed because it read nothing back from the key for the "member with index 2". Looking at , I notice that it's writing to the "member with index 0" and expecting to read it back from the "member with index 2". I'm curious, is each index supposed to represent a separate redis instance? What is the consistency model here?
/cc
https://k8s- Failed: [] PetSet [Slow] [Feature:PetSet] [] Deploy clustered applications [Slow] [Feature:PetSet] should creating a working redis cluster [Feature:PetSet] {Kubernetes e2e suite}
https://k8s- Failed: [] PetSet [Slow] [Feature:PetSet] [] Deploy clustered applications [Slow] [Feature:PetSet] should creating a working redis cluster [Feature:PetSet] {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
It's master slave redis, you should be able to create the same petset with this: , fyi
https://k8s- Failed: [] PetSet [Slow] [Feature:PetSet] [] Deploy clustered applications [Slow] [Feature:PetSet] should creating a working redis cluster [Feature:PetSet] {Kubernetes e2e suite}
https://k8s- Failed: [] PetSet [Slow] [Feature:PetSet] [] Deploy clustered applications [Slow] [Feature:PetSet] should creating a working redis cluster [Feature:PetSet] {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] PetSet [Slow] [Feature:PetSet] [] Deploy clustered applications [Slow] [Feature:PetSet] should creating a working redis cluster [Feature:PetSet] {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
Code change:
} By("Reading value under foo from member with index 2") <del> if v := pet.read(2, "foo"); v != "bar" { framework.Failf("Read unexpected value %v, expected bar under key foo", v) </del> <ins> if err := pollReadWithTimeout(pet, 2, "foo", "bar"); err != nil { framework.Failf("%v", err) </ins> } }) })

Related discussion:
text: https://k8s- Failed: [] PetSet [Slow] [Feature:PetSet] [] Deploy clustered applications [Slow] [Feature:PetSet] should creating a working redis cluster [Feature:PetSet] {Kubernetes e2e suite}
https://k8s- Failed: [] PetSet [Slow] [Feature:PetSet] [] Deploy clustered applications [Slow] [Feature:PetSet] should creating a working redis cluster [Feature:PetSet] {Kubernetes e2e suite}
https://k8s- Failed: [] PetSet [Slow] [Feature:PetSet] [] Deploy clustered applications [Slow] [Feature:PetSet] should creating a working redis cluster [Feature:PetSet] {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
Code change:
switch groupVersion { case extensionsv1beta1.SchemeGroupVersion: // no-op for compatibility <del> case appsv1beta2.SchemeGroupVersion: </del> <ins> case appsv1beta2.SchemeGroupVersion, appsv1.SchemeGroupVersion: </ins> // disallow mutation of selector allErrs = append(allErrs, apivalidation.ValidateImmutableField(newDaemonSet.Spec.Selector, oldDaemonSet.Spec.Selector, field.NewPath("spec").Child("selector"))...) default:

Related discussion:
text: /priority failing-test /priority critical-urgent /sig apps This test case has been failing for a while and affects a number of jobs: This is affecting multiple jobs on the testgrid dashboard, and prevents us from cutting 1.9.0-alpha.2 (kubernetes/sig-release). Is there work ongoing to bring this job back to green?
All test failures look similar: . ~Is it due to network issue instead of DaemonSet?~
This seems related to the new DaemonSet. The connection refused error is happening because kube-apiserver is crashing:
needs an update
Ah, well... I guess it's halfway working as intended. We put that there to make the tests fail if we forgot to update it when adding a new version. We had hoped it would fail during presubmit tests though. :(
Sending a fix now
I wonder why the test is not failing on submit queue tests. Do i need to fix deployment and replicaset?
The rest of the group is not merged, and the v1 PR takes the selector mutability into account.
Since this bug crashes apiserver I'd expect it to affect many other e2e tests.
Noted, we need to fix deployment and replicaset when the v1 PR is merged.
Not quite sure how this test isn't failing before but is now consistently failing. I'd expect the commit that triggers this to block the submit queue. The e2e test is still accessing DaemonSet through extensions/v1beta1 endpoints. From the above apiserver log it's hard to tell what updates DaemonSet via apps/v1 before the code panics. I tried to reproduce it locally, and found that garbage collector PUT apps/v1 DaemonSet:
When apps/v1 gets introduced, it may be ordered before apps/beta1 alphabetically. IIRC, there is a code change on the apps/v1 PR that changes the order of API versions such that apps/v1 is ranked before apps/v1beta2 and apps/v1beta2 before apps/v1beta1. Perhaps this problem can be fixed when the PR merges.
Since crashes apiserver and makes debugging harder, which defeats the purpose of using it, we should return error on unsupported versions instead.
The first run with this change is still in progress, but this test already passed: https://k8s- Thanks for the quick fix
Code change:
nodePortAddresses []string, kernelHandler KernelHandler, ) (*Proxier, error) { <del> // Set the route_localnet sysctl we need for if err := utilproxy.EnsureSysctl(sysctl, sysctlRouteLocalnet, 1); err != nil { return nil, err } </del> // Proxy needs br_netfilter and bridge-nf-call-iptables=1 when containers // are connected to a Linux bridge (but not SDN bridges).  Until most // plugins handle this, log when config is missing

Related discussion:
text: CVSS Rating: In typical clusters: medium (5.4) In clusters where API server insecure port has not been disabled: high (8.8) A security issue was discovered in which allows adjacent hosts to reach TCP and UDP services bound to running on the node or in the node's network namespace. For example, if a cluster administrator runs a TCP service on a node that listens on , because of this bug, that service would be potentially reachable by other hosts on the same LAN as the node, or by containers running on the same node as the service. If the example service on port required no additional authentication (because it assumed that only other localhost processes could reach it), then it could be vulnerable to attacks that make use of this bug. The Kubernetes API Server's default insecure port setting causes the API server to listen on where it will accept requests without authentication. Many Kubernetes installers explicitly disable the API Server's insecure port, but in clusters where it is not disabled, an attacker with access to another system on the same LAN or with control of a container running on the master may be able to reach the API server and execute arbitrary API requests on the cluster. This port is deprecated, and will be removed in Kubernetes v1.20. You may be vulnerable if: You are running a vulnerable version (see below) Your cluster nodes run in an environment where untrusted hosts share the same layer 2 domain (i.e. same LAN) as nodes Your cluster allows untrusted pods to run containers with (the Kubernetes default is to allow this capability). Your nodes (or hostnetwork pods) run any localhost-only services which do not require any further authentication. To list services that are potentially affected, run the following commands on nodes: - On a master node, an lsof entry like this indicates that the API server may be listening with an insecure port: kubelet/kube-proxy v1.18.0-1.18.3 kubelet/kube-proxy v1.17.0-1.17.6 kubelet/kube-proxy <=1.16.10 Prior to upgrading, this vulnerability can be mitigated by manually adding an iptables rule on nodes. This rule will reject traffic to 127.0.0.1 which does not originate on the node. Additionally, if your cluster does not already have the API Server insecure port disabled, we strongly suggest that you disable it.
Code change:
nodePortAddresses []string, kernelHandler KernelHandler, ) (*Proxier, error) { <del> // Set the route_localnet sysctl we need for if err := utilproxy.EnsureSysctl(sysctl, sysctlRouteLocalnet, 1); err != nil { return nil, err } </del> // Proxy needs br_netfilter and bridge-nf-call-iptables=1 when containers // are connected to a Linux bridge (but not SDN bridges).  Until most // plugins handle this, log when config is missing

Related discussion:
text: Add the following flag to your kubernetes API server command line: Packets on the wire with an IPv4 destination in the range 127.0.0.0/8 and a layer-2 destination MAC address of a node may indicate that an attack is targeting this vulnerability. Although the issue is caused by , the current fix for the issue is in (although future versions may have the fix in instead). We recommend updating both and to be sure the issue is addressed. The following versions contain the fix: kubelet/kube-proxy master - fixed by kubelet/kube-proxy v1.18.4+ - fixed by kubelet/kube-proxy v1.17.7+ - fixed by kubelet/kube-proxy v1.16.11+ - fixed by To upgrade, refer to the documentation: This issue was originally raised in issue which details how the sets which causes the system not to reject traffic to localhost which originates on other hosts. IPv6-only services that bind to a address are not affected. There may be additional attack vectors possible in addition to those fixed by and its cherry-picks. For those attacks to succeed, the target service would need to be UDP and the attack could only rely upon sending UDP datagrams since it wouldn't receive any replies. Finally, the target node would need to have reverse-path filtering disabled for an attack to have any effect. Work is ongoing to determine whether and how this issue should be fixed. See for up-to-date status on this issue. This vulnerability was reported by János Kövér, Ericsson with additional impacts reported by Rory McCune, NCC Group and Yuval Avrahami and Ariel Zelivansky, Palo Alto Networks. /area security /kind bug /committee product-security /sig network /sig node /area kubelet
For insecure port removal:
The original announcement contained an error. The announcement makes it sound like upgrading kube-proxy will address the issue, but that is incorrect. I have edited the issue summary to add the following paragraph in the "Fixed Versions" section: Additionally, I have changed the components in listed versions to say kubelet/kube-proxy instead of kube-proxy.
Could you add numerical CVSS scores to these? "High" is too vague, frequently varies from vendor to vendor. I can calculate a score from the vector myself, but adding the score would save us some work...
Edited.
Code change:
nodePortAddresses []string, kernelHandler KernelHandler, ) (*Proxier, error) { <del> // Set the route_localnet sysctl we need for if err := utilproxy.EnsureSysctl(sysctl, sysctlRouteLocalnet, 1); err != nil { return nil, err } </del> // Proxy needs br_netfilter and bridge-nf-call-iptables=1 when containers // are connected to a Linux bridge (but not SDN bridges).  Until most // plugins handle this, log when config is missing

Related discussion:
text: Also, the vector scores are clickable and will show the CVSS scores too.
To be clear, this is fixed in v1.18.4+, v1.17.7+, and v1.16.11+ correct?
Right. Sorry if the extra paragraph in "Fixed Versions" made that unclear.
Got it. Thank you!
So just to be sure, this wouldn't be a problem in an EKS cluster running in its own VPC, right? (assuming no untrusted hosts on the same VPC)
Are all the conditions listed under , a logical AND ?
Edit: after more testing, I think is indeed required
The attack can be done from any pod with (this is still K8S default), so you might not be safe. a more accurate title: "Node setting allows to bypass localhost boundary"
We believe that of the four conditions, you might be affected if (1 and (2 or 3) and 4) are true for your cluster. Sorry that it's not super clear from the text. Trying and failing to convey info concisely and with precision makes me glad that I'm in software, not law.
Isn't this broadly fixed by ? We're discussing better answers and options for deployment, too, but I am curious what the close-condition for this bug is?
I made a typo in the issue description saying that this is addressed by instead of . I'll edit it now. Sorry about that. Yes, this is mostly fixed by . I marked this as closed due to that PR fixing the majority of security issues caused by the setting. We can re-open if people think that we should address the remaining case that talks about.
Sorry, folks. I thought that this issue was already closed. Maybe that makes my response above to less confusing.
Yes, this being open was one of the major points of confusion for us. Appreciate the clarification
/label official-cve-feed (Related to )
Code change:
"path/filepath" "reflect" goruntime "runtime" <ins> "sort" </ins> "sync" "sync/atomic" "testing"

Related discussion:
text: TestUpdateAllocatedResourcesStatus Few days ago No response No response /sig node
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
I could reproduce this with in maybe 1/10 times.
cc I am unfamilar with this code so wanted to see if you have any ideas on this.
yes, I'll have a look ASAP
In the flake, the ResourceHealth array is with a different order. If this is acceptable, we should tune the test.
nice catch ! I think the best fix is ensure the ordering of statuses either in the production code or in the test code before to check Note: if we do this in the production code then we should go full throttle and offer a full API guarantee (docs, explicit tests) because users will depend on this
A quick fix is like if we want to ignore the order.
Code change:
import ( "context" <ins> "encoding/json" </ins> "fmt" "io/ioutil" "math/rand"

Related discussion:
text: <!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: Which test(s) are flaking: Testgrid link: Reason for failure: Unknown yet. Anything else we need to know: links to appreciated links to specific failures in spyglass appreciated /sig scheduling /help
Can I give it a try?
Sure, feel free to take it. /assign ahmad-diaa /remove-help
I keep trying to get it fail locally by running the following command: but it never does, am I doing something wrong? Is there something else that I should be doing and would be more helpful?
Sometimes the flake is hard to reproduce in local env. You may have to read the error log of the flakes thoroughly to get some clues. BTW: you can also run the test multiple times using the directive:
After looking through the logs I came up with some information that might be useful As far as I understand, the reason for the flake is the of the medium priority pod not being cleared That is because the update request for removing returns the following error: So my current guess is that might be outdated in some cases As I am not really familiar with the logic, I would like to know whether or not those are valid points so that I could dig deeper. What do you think?
If that's the case, you can set here: You can raise a PR and manually try for ~20 times to see you can reproduce the flake.
found another issue tracking this too. it seems it has been flaky for a while.
Sorry, didn't mean to take this one for you. I worked the duplicate issue and submitted a PR. Feel free to review that PR, or submit your own if you have a different solution.
This sometimes happens. But we usually take the first PR unless it's clearly worked by someone. In terms of which style to resolve the Conflict error, I'm asking here:
Sorry, didn't mean to take this one for you. I worked the duplicate issue and submitted a PR. Feel free to review that PR, or submit your own if you have a different solution. No worries, I haven't started implementing yet and your PR seems to be following the correct approach as well according to this: