jiebi/CodeConvo huggingface dataset:
Code change:
calls:        []string{"inspect_image", "pull"}, err:          true, }, <del> "should pull private image using dockerauth if image doesn't exist": { injectImage:  true, imgNeedsAuth: true, injectErr:    libdocker.ImageNotFoundError{ID: "image_id"}, calls:        []string{"inspect_image", "pull"}, configJSON:   authJSON, err:          false, }, </del> } { t.Logf("TestCase: %q", desc) _, fakeDocker, _ := newTestDockerService()

Related discussion:
text: Could you also verify whether fixed your problem?
tested against your commits cherry-picked on top of 1.8 beta, the fix seems to work
thanks.
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/node: Issue will be escalated to these SIGs if needed. priority/important-soon: Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details
Code change:
st.MakePod().Name("p1").UID("p1").Node("node1").Priority(midPriority).Obj(), st.MakePod().Name("p2").UID("p2").Node("node2").Priority(midPriority).Obj(), }, <del> expected: [][]Candidate{ { &candidate{victims: &extenderv1.Victims{}, name: "node1"}, }, }, expectedNumFilterCalled: []int32{3}, </del> <ins> expected:                [][]Candidate{{}}, fakeFilterRC:            framework.Unschedulable, expectedNumFilterCalled: []int32{2}, </ins> }, { name: "a pod that fits on both nodes when lower priority pods are preempted",

Related discussion:
text: <!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Kubernetes 1.19 (confirmed with 1.19.7 and 1.19.10) kube-scheduler crashes and restarts with the following errors in . panic: runtime error: index out of range [0] with length 0 The problem code is line 389. When is nil or is empty, the error happens. If we skip when it's nil or empty, the error is gone. panic: runtime error: invalid memory address or nil poster dereference The problem is does not exist and returns nil sometimes. Simply skipping it won't solve the problem. The scheduler will reach either line 456 or 341. The scheduler works without failures. When there are preemptions in a cluster. Kubernetes version (use ): v1.19.10 or v1.19.7 Cloud provider or hardware configuration: bare metal OS (e.g: ): centos 7 Kernel (e.g. ): Linux 5.4.77-7.el7pie SMP Sat Nov 21 01:16:27 UTC 2020 x8664 x8664 x8664 GNU/Linux Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling
/cc
/assign
Are you running the vanilla scheduler? or a customized one that leverages the utilities in
It's a custom scheduler with some internal out-of-tree plugins (just like ), but it doesn't have any custom postFilter plugins. Which utility functions are you referring to? Thanks.
are those custom plugins filter plugins? do they maintain state?
Code change:
st.MakePod().Name("p1").UID("p1").Node("node1").Priority(midPriority).Obj(), st.MakePod().Name("p2").UID("p2").Node("node2").Priority(midPriority).Obj(), }, <del> expected: [][]Candidate{ { &candidate{victims: &extenderv1.Victims{}, name: "node1"}, }, }, expectedNumFilterCalled: []int32{3}, </del> <ins> expected:                [][]Candidate{{}}, fakeFilterRC:            framework.Unschedulable, expectedNumFilterCalled: []int32{2}, </ins> }, { name: "a pod that fits on both nodes when lower priority pods are preempted",

Related discussion:
text: If yes, then one hypothesis is that those custom filters filter the node in the filter phase, but not when executed in the preemption phase, and so this will result in a candidate node with no victims?
Yes, it includes filter plugins and uses cycle state.
you need to make sure that those filter plugins play nicely when executed again in the preemption phase in the same cycle: i.e., produce the same result.
Thanks What do you mean by performing filter in the preemption phase? would you mind elaborating it a little?
What additional logic is needed to handle the preemption case?
in the preemption phase we run the filters again to check if removing lower priority pods make the node schedulable: we later add them one by one to reduce the set of victim pods to the absolute minimum: My hypothesis is that somehow your filter returns success when adding all the pods back, and so you end up with zero victims and a candidate node. This shouldn't happen because if the pod fits the node without removing any victims, then we shouldn't be running preemption in the first place. So the theory is that the custom filter is returning false when run in the filter phase, but returns true when executed in the preemption phase perhaps because it makes some assumptions about cyclestate that makes it behave this way (e.g. that a the filter will be executed once per node in a scheduling cycle).
Disabled the filter plugin, but still see the same issues.
are you able to change the scheduler code and run the test again? I can send a patch tomorrow to add some debugging messages to help us root cause the issue.
yes, thanks a lot! really appreciate it!
You are right. We still use a deprecated Predicate extender, which causes the problem. After disabling it, it's working fine. It used to work with 1.18 though. We are retiring it. Thank you so much!!!
If the test cases can help debug scheduler extender Webhook (Predicate) too, I'd like to try it. Thanks again!
How can we improve the preemption logic to prevent bugs and issues in custom Filters or Predicates crashing the scheduler?
for starter, we should check the list len, the scheduler should not panic in all cases.
Code change:
st.MakePod().Name("p1").UID("p1").Node("node1").Priority(midPriority).Obj(), st.MakePod().Name("p2").UID("p2").Node("node2").Priority(midPriority).Obj(), }, <del> expected: [][]Candidate{ { &candidate{victims: &extenderv1.Victims{}, name: "node1"}, }, }, expectedNumFilterCalled: []int32{3}, </del> <ins> expected:                [][]Candidate{{}}, fakeFilterRC:            framework.Unschedulable, expectedNumFilterCalled: []int32{2}, </ins> }, { name: "a pod that fits on both nodes when lower priority pods are preempted",

Related discussion:
text: Other than that, I think we need to clearly document what the preemption logic does (that it executes the prefilter extensions points and the filter plugins multiple times) and the expectations from filter plugins (that they may get executed more than once in the same cycle)
Is it a predicate extender or preemption extender? If it's a predicate extender, I don't quite think we found the root cause - b/c predicate extenders are not invoked during preemption (). It makes more sense if it's a preemption extender - if the preemption extender mutates the to make it invalid. If it's the case, could you verify it by adding debug info before and after L157 to check the validness of ?
That's always true :)
filed a PR to mitigate it.
It's a predicate extender. I'll debug it more.
Added some debug info. after and . Here's the result. returned 4 candidates, but one of candidates () has zero pod. The candidate happened to be the best candidate and hence caused the "" error. It should have nothing to do with since our extender doesn't support Preemption. Disabling it may change pod placements/candidates selection and bypassed the error. It's like a coincidence. To remove the panic, a simple fix is to check if is empty. If it is, do not add the node to candidates. Alternatively, a change to may work too. Here is the log with debug info.
This looks more promising. Back to digging into the root cause, it seems returns an empty but with . So that implies firstly we have a non-nil , and then in , the is separated into two list: violatingVictims and nonViolatingVictims. Finally the two victims list are tried one by one to run . It looks like returns {fits=true, nil error} for every victim, which is quite abnormal. I think it's problematic that every pod is retrievable and in which we can debug into.
Yes, should not return fit=true with empty pods in the first place, but it's hard to locate the problematic code in this function.
The following code in looks questionable to me. If is empty, should the function return nil, nil? Also, if is true for all extenders, the loop won't do anything and candidates will remain the same.
Code change:
st.MakePod().Name("p1").UID("p1").Node("node1").Priority(midPriority).Obj(), st.MakePod().Name("p2").UID("p2").Node("node2").Priority(midPriority).Obj(), }, <del> expected: [][]Candidate{ { &candidate{victims: &extenderv1.Victims{}, name: "node1"}, }, }, expectedNumFilterCalled: []int32{3}, </del> <ins> expected:                [][]Candidate{{}}, fakeFilterRC:            framework.Unschedulable, expectedNumFilterCalled: []int32{2}, </ins> }, { name: "a pod that fits on both nodes when lower priority pods are preempted",

Related discussion:
text: Why do we bother creating a new candidates later? Can it just return candidates?
It can prevent from returning candidates with empty pods, but what about ? It's possible to add nodes without victim pods to candidates if returned by contain such nodes. How about adding the following check to CallExtender?
Updated the PR based on the finding and discussions.
Yes, we can use a flag to mark if all extenders don't support preemption. If yes, simply return the candidates.
Technically this can exist due to faulty extender implementation. If we really want to guard it, instead of logging the error and continue, I'm more inclined to return the error immediately as this is a fatal error - the victimsMap cannot be used either for latter extender or further preemptor nominating.
Is my understanding correct? In , if victimsMap is empty, should it return instead of ?
The only case victimsMap can be empty is empty candidates, and so return either doesn't quite matter, right?
You are right.
As long as one extender returns an invalid victimsMap (with empty pods), s returns an error immediately? What about ? Should it return an error immediately if it returnswith empty pods?
I think so. Because the result it returned is faulty, and we don't want to continue based on it The same. We may don't want to continue the scheduling cycle based on faulty result.
Code change:
"//staging/src/k8s.io/client-go/kubernetes/fake:go_default_library", "//staging/src/k8s.io/client-go/testing:go_default_library", "//staging/src/k8s.io/client-go/tools/clientcmd:go_default_library", <del> "//staging/src/k8s.io/cluster-bootstrap/token/api:go_default_library", </del> "//vendor/github.com/ghodss/yaml:go_default_library", "//vendor/github.com/pkg/errors:go_default_library", "//vendor/github.com/renstrom/dedent:go_default_library",

Related discussion:
text: Just got this while running the tests:
let's please move this to /close /sig cluster-lifecycle
Code change:
return &d, nil } <ins> // getIPFamily parses the IP family based on raw data from netlink. // For AF_INET, netlink will set the first 4 bytes with trailing zeros //   10.0.0.1 -> [10 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0] // For AF_INET6, the full 16 byte array is used: //   2001:db8:3c4d:15::1a00 -> [32 1 13 184 60 77 0 21 0 0 0 0 0 0 26 0] func getIPFamily(address []byte) (uint16, error) { if len(address) == 4 { return syscall.AF_INET, nil } if isZeros(address) { return 0, errors.New("could not parse IP family from address data") } // assume IPv4 if first 4 bytes are non-zero but rest of the data is trailing zeros if !isZeros(address[:4]) && isZeros(address[4:]) { return syscall.AF_INET, nil } return syscall.AF_INET6, nil } func isZeros(b []byte) bool { for i := 0; i < len(b); i++ { if b[i] != 0 { return false } } return true } </ins> // parseDestination given a ipvs netlink response this function will respond with a valid destination entry, an error otherwise func (i *Handle) parseDestination(msg []byte) (*Destination, error) { var dst *Destination

Related discussion:
text: Hi, Since my upgrade to 1.18 version, I have errors in kube-proxy: I have ipv4/ipv6 dualstack enable. No problem with cluster and IPVS works despite errors. Do you have also this issue ? OS: RHEL7 Kubernetes version (use ):
/sig network
Related to cc
/cc
Taking one of the outputs it gives me the following addresses? can you share more details about your environment, Is it possible that the CoreDNS pod has the address ? what are your cluster-cidr and service-cidr? Indeed that new net.Utils change looks suspicious , a quick grep in the repo gives me some suspects, maybe the service/ipallocator? It's important to mention there were similar issues with IPVS before, interpreting IPv4 addresses as IPv6, has some experience with this kind of issues, maybe he can see something:) /cc
A bit tricky to trace since libnetwork/ipvs is moving around at the moment. Problem seems to be in; The "family" must be wrong. It is called from;
Please do a and attach the output.
Hi, Some informations: cluster-cidr and service-cidr: fd00:10:245:0:0:0:0:0/64,10.245.0.0/16 fd00:10:96:0:0:0:0:0/112,10.96.0.0/12 ipvsadm output:
Ref:
The commit in the ref above should be in v1.18.0 which is the version used in this issue. So the commit does not fix the problem.
Maybe something wrong around here Before , it looks like this: Now it looks like this: However is not initializedit is zero :(.
I do not get these faults. Has the netlink API changed? What kernel version are you using?
d.AddressFamily is supposed to be set from the "NetlinkRouteAttr" slice; It should definitely not be hard-coded to ipv4.
Made a trying to reproduce this in my environment but didn't got this error.
Code change:
return &d, nil } <ins> // getIPFamily parses the IP family based on raw data from netlink. // For AF_INET, netlink will set the first 4 bytes with trailing zeros //   10.0.0.1 -> [10 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0] // For AF_INET6, the full 16 byte array is used: //   2001:db8:3c4d:15::1a00 -> [32 1 13 184 60 77 0 21 0 0 0 0 0 0 26 0] func getIPFamily(address []byte) (uint16, error) { if len(address) == 4 { return syscall.AF_INET, nil } if isZeros(address) { return 0, errors.New("could not parse IP family from address data") } // assume IPv4 if first 4 bytes are non-zero but rest of the data is trailing zeros if !isZeros(address[:4]) && isZeros(address[4:]) { return syscall.AF_INET, nil } return syscall.AF_INET6, nil } func isZeros(b []byte) bool { for i := 0; i < len(b); i++ { if b[i] != 0 { return false } } return true } </ins> // parseDestination given a ipvs netlink response this function will respond with a valid destination entry, an error otherwise func (i *Handle) parseDestination(msg []byte) (*Destination, error) { var dst *Destination

Related discussion:
text: I may try this on some older kernel like from CentOS 7 Added exactly the same IPVS configuration here: And when running this 'gist/program' pointing to the IPVS from the DNS (fd00:10:96::a port 53 UDP) got the expected result: Maybe trying to run this also in some other scenarios could clarify if this is something related to Netlink API changed between Kernels, something with the used library, etc.
After some study, I find that this issue most likely is caused by the low linux kernel version. I have reproduced this problem on . Accroding to the code, netlink will try to get attribute, but I find the kernel just does not support it :( This is the defined in (kernel 3.10) No attribute is defined! But in new kernel version, the is defined like : Obviously, the kernel has some attributes (IPVSDESTATTRADDRFAMILY, IPVSDESTATTRSTATS64...). That is why kube-proxy works well on systems with a higher version of the kernel. So we can address this issue by upgrading our linux kernel. Not sure which kernel version is the minimum requirement, maybe we should document something about this.
Here is the defined in :
Good finding. So, we can conclude that this really is kernel version dependent. It is hard to know what to do about it. We must check the family. I would propose that we make it work for ipv4 with some fallback and put a requirement on kernel version for ipv6-only/dual-stack. But the decision is others to make. /cc
/cc
The symbol in the kernel tree The for service family seems very old. Since the service must have the same family as the real-servers perhaps it can be used for compatibility with old kernels?
The commit that adds the symbol; Seem to go into linux-3.18
I did not know the minimum required for the kernel version, I was using the stable and official version of RHEL7. My kernel version is: Currently the max version in the repository is: According to the documentation (), I should switch to RHEL8 if I want a recent kernel :-
There is no minimum kernel version requirement afaik. So I think this must be fixed for ipv4-only clusters IMO.
Code change:
return &d, nil } <ins> // getIPFamily parses the IP family based on raw data from netlink. // For AF_INET, netlink will set the first 4 bytes with trailing zeros //   10.0.0.1 -> [10 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0] // For AF_INET6, the full 16 byte array is used: //   2001:db8:3c4d:15::1a00 -> [32 1 13 184 60 77 0 21 0 0 0 0 0 0 26 0] func getIPFamily(address []byte) (uint16, error) { if len(address) == 4 { return syscall.AF_INET, nil } if isZeros(address) { return 0, errors.New("could not parse IP family from address data") } // assume IPv4 if first 4 bytes are non-zero but rest of the data is trailing zeros if !isZeros(address[:4]) && isZeros(address[4:]) { return syscall.AF_INET, nil } return syscall.AF_INET6, nil } func isZeros(b []byte) bool { for i := 0; i < len(b); i++ { if b[i] != 0 { return false } } return true } </ins> // parseDestination given a ipvs netlink response this function will respond with a valid destination entry, an error otherwise func (i *Handle) parseDestination(msg []byte) (*Destination, error) { var dst *Destination

Related discussion:
text: However ipv6-only and dual-stack are in "alpha" so a requirement on kernel version might be ok, but I can't say myself. A complication is that the problem is not in k8s itself but in a 3rd party lib.
ipv6-only is beta since 1.18 :smile:
/assign
I opened a PR to fix this in moby/ipvs If someone w/ a cluster on the older kernel version can test it that would be great, otherwise I will setup a cluster to validate this some time this week.
Instead of bootstraping a whole cluster right now (pretty busy here) I can use the Gist I've mentioned to check in a CentOS 7.1 if it faces the problem before and after the patch. At least to have some more test right now
with a small test, it worked fine. Environment: CentOS 7.1 - Kernel 3.10.0-229.el7.x86_64 + ipvsadm 1.2.1 Pre-loaded some IPv6 rules into ipvs: I've used code, searching for the following structure: I then call ` to get the IPVS object, and then try to fetch the destinations. The following are the results: Before patching the ipvs module: After patching the ipvs module (pinning to your repo/branch) Also I've made those tests with the structure but searching for TCP, and for kube-apiserver (that have only one destination) and it worked fine in all those scenarios. If wanted I can improve the gist to pass the parameters as command arguments instead of hardcoded (sorry, did this in a rush to test)
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry. ‍
/remove-triage unresolved
Thanks just waiting on now
same issue,sloved by update kernel to 5.X
I didn't want to wait, I updated to kernel-lt in the version 4.4 . The problem is solved on my side :)
Thanks for confirming. We'll still try to get in since I'm sure other folks with older kernels will run into this issue.
/retitle parseIP error with IPVS proxy
upgrade your system kernel to 4.xx edition
FYI: just merged, need to run some validation & cut a new release (v1.0.1).
Code change:
return &d, nil } <ins> // getIPFamily parses the IP family based on raw data from netlink. // For AF_INET, netlink will set the first 4 bytes with trailing zeros //   10.0.0.1 -> [10 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0] // For AF_INET6, the full 16 byte array is used: //   2001:db8:3c4d:15::1a00 -> [32 1 13 184 60 77 0 21 0 0 0 0 0 0 26 0] func getIPFamily(address []byte) (uint16, error) { if len(address) == 4 { return syscall.AF_INET, nil } if isZeros(address) { return 0, errors.New("could not parse IP family from address data") } // assume IPv4 if first 4 bytes are non-zero but rest of the data is trailing zeros if !isZeros(address[:4]) && isZeros(address[4:]) { return syscall.AF_INET, nil } return syscall.AF_INET6, nil } func isZeros(b []byte) bool { for i := 0; i < len(b); i++ { if b[i] != 0 { return false } } return true } </ins> // parseDestination given a ipvs netlink response this function will respond with a valid destination entry, an error otherwise func (i *Handle) parseDestination(msg []byte) (*Destination, error) { var dst *Destination

Related discussion:
text: FYI
/retitle DualStack / IPv6-only: parseIP error with IPVS proxy on CentOS 7
v1.18 cherry-pick
Hi, it seems that you're using v1.18.0 and this has been corrected in v1.18.3, can you please update and check? Tks
Code change:
stopCh := make(chan struct{}) fakeClientset := fake.NewSimpleClientset(&v1.PodList{Items: item.pods}) holder := &podHolder{} <del> controller := NewNoExecuteTaintManager(fakeClientset, holder.getPod, (&nodeHolder{item.newNode}).getNode, getPodsAssignedToNode(fakeClientset)) </del> <ins> controller := NewNoExecuteTaintManager(fakeClientset, holder.getPod, (&nodeHolder{node: item.newNode}).getNode, getPodsAssignedToNode(fakeClientset)) </ins> controller.recorder = testutil.NewFakeRecorder() go controller.Run(stopCh)

Related discussion:
text: What happened: A Pod was scheduled for deletion because a taint w/ effect was observed in the Node the Pod was assigned to, and after the taint is removed from the Node, the Pod scheduled deletion isn't canceled. After scheduling pod deletion, the only cancels pod deletion if all taints are removed from the node, including any taints that the user relies on for their use-cases (example below). What you expected to happen: After the taint that triggered scheduling pod deletion is removed, cancels the scheduled pod deletion. How to reproduce it (as minimally and precisely as possible): A node is tainted w/ , so any workloads already assigned to it which don't tolerate this taint are evicted. A pod is created w/ tolerations: w/ unspecified and w/ . is assigned to . At some point in time after the previous steps took place, is tainted as . gets an update event for and observes two taints. It proceeds to calculate the minimum time (in seconds) that tolerates for the two taints which at this point is 300 seconds, and marks for deletion in ~300 seconds - this happens in-memory, in a timed-queue. At some point in time after the previous steps took place, and before the 300 seconds trigger happen, taint is removed from . gets an update event for and observes only one taint, . It proceeds to calculate the minimum time (in seconds) that tolerates for the this taint which is infinity, and returns, not canceling the previous deletion. It completely ignores the fact that the taint that triggered deletion is no longer observed. Anything else we need to know?: Worked w/ to figure this out after all pods in a node got evicted without apparent reason. Environment: Kubernetes version (use ): 1.17.3 (but looking at the code seems to affect all of 1.17.x and 1.18.x and master. Cloud provider or hardware configuration: N/A OS (e.g: ): N/A Kernel (e.g. ): 5.x Install tools: N/A Network plugin and version (if this is a network-related bug): N/A Others: N/A cc (owners of this code)
There seems to be at least .