jiebi/CodeConvo huggingface dataset:
Code change:
} return conditionStatuses } <del> return !reflect.DeepEqual(strip(oldNode.Status.Conditions), strip(newNode.Status.Conditions)) </del> <ins> return !equality.Semantic.DeepEqual(strip(oldNode.Status.Conditions), strip(newNode.Status.Conditions)) </ins> } func nodeSpecUnschedulableChanged(newNode *v1.Node, oldNode *v1.Node) bool { return newNode.Spec.Unschedulable != oldNode.Spec.Unschedulable && !newNode.Spec.Unschedulable } <ins> func nodeAnnotationsChanged(newNode *v1.Node, oldNode *v1.Node) bool { return !equality.Semantic.DeepEqual(oldNode.GetAnnotations(), newNode.GetAnnotations()) } </ins> func preCheckForNode(nodeInfo *framework.NodeInfo) queue.PreEnqueueCheck { // Note: the following checks doesn't take preemption into considerations, in very rare // cases (e.g., node resizing), "pod" may still fail a check but preemption helps. We deliberately

Related discussion:
text: Let me summarize each: Born from from is right. But, that's the problem with preCheck. (Some preCheck depends on some in-tree plugins' logic and it affects badly to other plugins enqueueing ) The preCheck is removed after is completed. After preCheck is removed, will be the correct behavior. So, we can do the optimization that changes node to a specific event. will be fixed in: It's kind of rare case, but I feel we need to have some improvement like Edit: create another issue for this: I'd say (2) and (3) are kind of rare cases. So, the blocker here is (1). (And, (1) is blocked by the removal of preCheck, which is blocked by ) As my conclusion, it's safer to add after (1) is done so that we can make sure the performance won't be affected badly. what do you think?
Instead, can we just keep an event for "node updated", then individual plugins can decide whether a annotations change is worth requeuing or not?
Yes, (1) can be replaced with that idea. But, I'd say changing to a specific is slightly better because the scheduling queue could filter out some Node update events without executing the hint functions.
If rescheduling can be triggered with precise events, that would be great. I haven't revisit since last time, I'll take a look again.
Since we are missing an event for , can we add it? But we also have to modify existing events like so: Because change of such field is also an Update. And in future releases we can get rid of some fine-grained events.
Since we will add hintFuncs for all in-tree plugins, so I think it's ok to add nodeAnnotationsChanged event now, we can filter out unrelated events in hitFuncs. Any other concerns? cc Prefer to keep them as people can register fine-gained events instead a general event.
If by "now" you mean v1.30, then I'm ok with that.
Code change:
//    (localhost) by name. The possible profile names are detailed at //    http://wiki.apparmor.net/index.php/AppArmor_Core_Policy_Reference ApparmorProfile string `protobuf:"bytes,9,opt,name=apparmor_profile,json=apparmorProfile,proto3" json:"apparmor_profile,omitempty"` <ins> // Seccomp profile for the container, candidate values are: // * runtime/default: the default profile for the container runtime // * unconfined: unconfined profile, ie, no seccomp sandboxing // * localhost/<full-path-to-profile>: the profile installed on the node. //   <full-path-to-profile> is the full path of the profile. SeccompProfilePath string `protobuf:"bytes,10,opt,name=seccomp_profile_path,json=seccompProfilePath,proto3" json:"seccomp_profile_path,omitempty"` </ins> } func (m *LinuxContainerSecurityContext) Reset()      { *m = LinuxContainerSecurityContext{} }

Related discussion:
text: CRI supports AppArmor, Seccomp and Sysctls via annotations , we should move them to typed and structured fields instead. (refer ) [x] Apparmor () [x] Sysctls () [x] Seccomp () cc
/cc
/sig node
Seccomp is addressed at .
Code change:
//    (localhost) by name. The possible profile names are detailed at //    http://wiki.apparmor.net/index.php/AppArmor_Core_Policy_Reference ApparmorProfile string `protobuf:"bytes,9,opt,name=apparmor_profile,json=apparmorProfile,proto3" json:"apparmor_profile,omitempty"` <ins> // Seccomp profile for the container, candidate values are: // * runtime/default: the default profile for the container runtime // * unconfined: unconfined profile, ie, no seccomp sandboxing // * localhost/<full-path-to-profile>: the profile installed on the node. //   <full-path-to-profile> is the full path of the profile. SeccompProfilePath string `protobuf:"bytes,10,opt,name=seccomp_profile_path,json=seccompProfilePath,proto3" json:"seccomp_profile_path,omitempty"` </ins> } func (m *LinuxContainerSecurityContext) Reset()      { *m = LinuxContainerSecurityContext{} }

Related discussion:
text: Seccomp profile is passed to runtime by in CRI: is relative to node's local seccomp profile root, it is defined in kubelet. But runtime doesn't know it. We should pass full profile path in CRI. cc/ Will file a PR to fix this later.
Add support for Seccomp sounds like a new feature. The milestone should be reset to , correct?
not really. The current implementation supports seccomp already. If it's missing critical information to function properly, this would be a bug.
are we blocking releases for CRI in 1.5? i was not sure, so marking this as non-release-blocker and leaving 1.5 milestone. please correct if i am mistaken.
I don't get why this is a 1.5 targetted thing myself. Yes we want CRI to be in a good state in 1.5, but afaik we have zero intention of anyone actually using (at the very least because of the lack of exec auth regressing kubelet security). why should this be part of 1.5?
First, this is a small enough change that allows non-docker integrations to properly support seccomp. Second, if "we have zero intention of anyone actually using" is a valid reason, we should never fix any bugs in CRI. We do want people to try using it (perhaps not in production yet). However, given that seccomp is only in alpha, I am ok with dropping this and just continue the docker support for now.
Perhaps I misunderstood the meaning of code freeze, but I thought it was only bug-fixes regardless of size of change (or exceptions that have gone through some process) What I meant is we don't intend this to be used generally as part of the 1.5 release. Because of that, I don't think we should consider bugs in CRI bugs for the purpose of the 1.5 release freeze. My statement is just that we shouldn't violate code-freeze for this. We should totally fix this right after the 1.5 freeze window ends, and saying that I'm arguing we should never fix anything is disingenuous and taking it to an absurd extreme.
the only thing we don't seem to agree on is whether seccomp support was considered to be supported for CRI in 1.5.
Code change:
//    (localhost) by name. The possible profile names are detailed at //    http://wiki.apparmor.net/index.php/AppArmor_Core_Policy_Reference ApparmorProfile string `protobuf:"bytes,9,opt,name=apparmor_profile,json=apparmorProfile,proto3" json:"apparmor_profile,omitempty"` <ins> // Seccomp profile for the container, candidate values are: // * runtime/default: the default profile for the container runtime // * unconfined: unconfined profile, ie, no seccomp sandboxing // * localhost/<full-path-to-profile>: the profile installed on the node. //   <full-path-to-profile> is the full path of the profile. SeccompProfilePath string `protobuf:"bytes,10,opt,name=seccomp_profile_path,json=seccompProfilePath,proto3" json:"seccomp_profile_path,omitempty"` </ins> } func (m *LinuxContainerSecurityContext) Reset()      { *m = LinuxContainerSecurityContext{} }

Related discussion:
text: My interpretation is that we assumed seccomp would be supported until we found out that it wouldn't for other runtimes due to the lack of profile path. That's the only reason why I considered this a bug. Of course we could say that this is not really an important feature in CRI, so we can drop it, but we can be more clear on that.
If there's a bug in a feature no one uses, does fixing it make a sound? I think what we're not agreeing on is the role of CRI in 1.5. I understood it as something that is not generally usable. We can't recommend a customer use it, even as an experimental feature, due to security aspects (if nothing else). Correct me if I'm wrong. It's a developer integration point, and developers integrating with it are doing so with the kubernetes master branch, not with some release. Those reasons together result in my opinion that CRI is still not really a customer/release feature, and thus bugs or features related to it should not be exceptions to the freeze window, but rather should targeted at master as soon as the freeze lifts. If this were already post-freeze, would we be cherry-picking this over to 1.5?
This is not true. cri-o is depending on this fix for apparmor support.
is it depending on the 1.5 release branch directly, or on master?
I think it's better to make this in 1.5, so cri-o could work with first release with experimental CRI. It helps to encourage people to use CRI and find features misses or bugs.
Modifying the user-specified annotation is a little bit unsettling. I think adding a new annotation explicitly for the path would be better. Or, we should just add an explicit field for this. I spoke to and separately. I still think this is a bug fix, but we may want to be more careful about the fix. Given the time constraint, let's just list this as a known issue and work on it for 1.6.
OK, let's move this to next release and think carefully about seccomp feature besides this: kubelet may just pass the profile as string via CRI; may assume the profile locate in a file in a known format and pass path to the runtime; or maybe some other solutions.
Code change:
//    (localhost) by name. The possible profile names are detailed at //    http://wiki.apparmor.net/index.php/AppArmor_Core_Policy_Reference ApparmorProfile string `protobuf:"bytes,9,opt,name=apparmor_profile,json=apparmorProfile,proto3" json:"apparmor_profile,omitempty"` <ins> // Seccomp profile for the container, candidate values are: // * runtime/default: the default profile for the container runtime // * unconfined: unconfined profile, ie, no seccomp sandboxing // * localhost/<full-path-to-profile>: the profile installed on the node. //   <full-path-to-profile> is the full path of the profile. SeccompProfilePath string `protobuf:"bytes,10,opt,name=seccomp_profile_path,json=seccompProfilePath,proto3" json:"seccomp_profile_path,omitempty"` </ins> } func (m *LinuxContainerSecurityContext) Reset()      { *m = LinuxContainerSecurityContext{} }

Related discussion:
text: We need to clearly define it. define profile format clearly
Please document known issues for 1.5 in
(copied from ) And let's move here to discuss how to pass profiles to runtime. Related with .
This is issue is related with two level APIs: CRI, since we may also want to support profiles from ConfigMaps, I think we should pass full profile contents instead of path to runtime. issues is also related with seccomp api in kube-apiserver, it is based on annotations today. We should define seccomp profile format clearly (see ) and decide whether to add struct types to pod spec or just continue the annotations way. I think it's more clear to add specific struct types to pod spec.
Agreed - the path stuff is really a crutch (and implementation-specific) - I think is the answer here.
I don't disagree, but it's not strictly a requirement for supporting ConfigMaps (kubelet could create a file with the contents). An open question is whether to pass it as a raw string, or wait for and pass the structured data. The annotations way is deprecated (). I think the bigger question is where/how to put the data. E.g. structured data in the API, unstructured (string type) data in the API, reference to another object (e.g. ConfigMap). If the Kubelet needs to parse the spec, I think that's a good argument for a structured data type (either a standalone object or embedded in the pod API).
Yes, already opened to replace annotations with typed structs. I personally prefer a structured data type in both kubernetes API and CRI. WDYT?
A counter argument is that as the spec grows or new features are , we need to keep updating the Kubernetes API to maintain parity, or update the code in each CRI implementation to translate the Kubernetes spec. OTOH, if we use unstructured data we can use 3rd party libraries to parse out the data we need (assuming the spec evolves in a backwards compatible way).
What spec?
Whichever seccomp profile spec is being implemented by runtimes. E.g. if we decide to base the Kubernetes seccomp profile API off the OCI seccomp profile spec, and OCI adds some new features, at some point we'd probably want to add those features to Kubernetes.
Code change:
//    (localhost) by name. The possible profile names are detailed at //    http://wiki.apparmor.net/index.php/AppArmor_Core_Policy_Reference ApparmorProfile string `protobuf:"bytes,9,opt,name=apparmor_profile,json=apparmorProfile,proto3" json:"apparmor_profile,omitempty"` <ins> // Seccomp profile for the container, candidate values are: // * runtime/default: the default profile for the container runtime // * unconfined: unconfined profile, ie, no seccomp sandboxing // * localhost/<full-path-to-profile>: the profile installed on the node. //   <full-path-to-profile> is the full path of the profile. SeccompProfilePath string `protobuf:"bytes,10,opt,name=seccomp_profile_path,json=seccompProfilePath,proto3" json:"seccomp_profile_path,omitempty"` </ins> } func (m *LinuxContainerSecurityContext) Reset()      { *m = LinuxContainerSecurityContext{} }

Related discussion:
text: It also means that if different runtimes support different feature sets, we'll be stuck either tracking only the common features, or specifying features that may be ignored by the runtime. This may be a non-issue if the seccomp feature set rarely changes.
Well, yes, but that applies to everything in the CRI, and is one of the tradeoffs we ma[d|k]e by introducing this abstraction in the first place.
My $0.02: Moving the existing logic/information for seccomp into a first-class field for CRI (as in ) is productive because of the known problems with the annotation approach. My initial proposal for seccomp support included a resource similar to what has proposed in kubernetes/community and we were unable to come to an agreement on the issue refers to here: I do not necessarily expect that we will be able to come to an agreement about the seccomp resource in time for 1.7, so I think we should proceed with and do backward compatible CRI changes later if we need to inject the content of the seccomp profile instead of the path. I think we would deeply regret embedding API surface to describe seccomp profiles in the Pod API :)
Code change:
expected       []string // any of the items is valid }{ { <del> name:           "No node needs preemption", registerPlugin: st.RegisterPluginAsExtensions(noderesources.FitName, noderesources.NewFit, "Filter", "PreFilter"), nodeNames:      []string{"node1"}, pod:            st.MakePod().Name("p").UID("p").Priority(highPriority).Req(largeRes).Obj(), pods: []*v1.Pod{ st.MakePod().Name("p1").UID("p1").Node("node1").Priority(midPriority).Req(smallRes).StartTime(epochTime).Obj(), }, expected: []string{"node1"}, }, { </del> name:           "a pod that fits on both nodes when lower priority pods are preempted", registerPlugin: st.RegisterPluginAsExtensions(noderesources.FitName, noderesources.NewFit, "Filter", "PreFilter"), nodeNames:      []string{"node1", "node2"},

Related discussion:
text: <!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Kubernetes 1.19 (confirmed with 1.19.7 and 1.19.10) kube-scheduler crashes and restarts with the following errors in . panic: runtime error: index out of range [0] with length 0 The problem code is line 389. When is nil or is empty, the error happens. If we skip when it's nil or empty, the error is gone. panic: runtime error: invalid memory address or nil poster dereference The problem is does not exist and returns nil sometimes. Simply skipping it won't solve the problem. The scheduler will reach either line 456 or 341. The scheduler works without failures. When there are preemptions in a cluster. Kubernetes version (use ): v1.19.10 or v1.19.7 Cloud provider or hardware configuration: bare metal OS (e.g: ): centos 7 Kernel (e.g. ): Linux 5.4.77-7.el7pie SMP Sat Nov 21 01:16:27 UTC 2020 x8664 x8664 x8664 GNU/Linux Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling
/cc
/assign
Are you running the vanilla scheduler? or a customized one that leverages the utilities in
It's a custom scheduler with some internal out-of-tree plugins (just like ), but it doesn't have any custom postFilter plugins. Which utility functions are you referring to? Thanks.
are those custom plugins filter plugins? do they maintain state?
Code change:
expected       []string // any of the items is valid }{ { <del> name:           "No node needs preemption", registerPlugin: st.RegisterPluginAsExtensions(noderesources.FitName, noderesources.NewFit, "Filter", "PreFilter"), nodeNames:      []string{"node1"}, pod:            st.MakePod().Name("p").UID("p").Priority(highPriority).Req(largeRes).Obj(), pods: []*v1.Pod{ st.MakePod().Name("p1").UID("p1").Node("node1").Priority(midPriority).Req(smallRes).StartTime(epochTime).Obj(), }, expected: []string{"node1"}, }, { </del> name:           "a pod that fits on both nodes when lower priority pods are preempted", registerPlugin: st.RegisterPluginAsExtensions(noderesources.FitName, noderesources.NewFit, "Filter", "PreFilter"), nodeNames:      []string{"node1", "node2"},

Related discussion:
text: If yes, then one hypothesis is that those custom filters filter the node in the filter phase, but not when executed in the preemption phase, and so this will result in a candidate node with no victims?
Yes, it includes filter plugins and uses cycle state.
you need to make sure that those filter plugins play nicely when executed again in the preemption phase in the same cycle: i.e., produce the same result.
Thanks What do you mean by performing filter in the preemption phase? would you mind elaborating it a little?
What additional logic is needed to handle the preemption case?
in the preemption phase we run the filters again to check if removing lower priority pods make the node schedulable: we later add them one by one to reduce the set of victim pods to the absolute minimum: My hypothesis is that somehow your filter returns success when adding all the pods back, and so you end up with zero victims and a candidate node. This shouldn't happen because if the pod fits the node without removing any victims, then we shouldn't be running preemption in the first place. So the theory is that the custom filter is returning false when run in the filter phase, but returns true when executed in the preemption phase perhaps because it makes some assumptions about cyclestate that makes it behave this way (e.g. that a the filter will be executed once per node in a scheduling cycle).
Disabled the filter plugin, but still see the same issues.
are you able to change the scheduler code and run the test again? I can send a patch tomorrow to add some debugging messages to help us root cause the issue.
yes, thanks a lot! really appreciate it!
You are right. We still use a deprecated Predicate extender, which causes the problem. After disabling it, it's working fine. It used to work with 1.18 though. We are retiring it. Thank you so much!!!
If the test cases can help debug scheduler extender Webhook (Predicate) too, I'd like to try it. Thanks again!
How can we improve the preemption logic to prevent bugs and issues in custom Filters or Predicates crashing the scheduler?
for starter, we should check the list len, the scheduler should not panic in all cases.
Code change:
expected       []string // any of the items is valid }{ { <del> name:           "No node needs preemption", registerPlugin: st.RegisterPluginAsExtensions(noderesources.FitName, noderesources.NewFit, "Filter", "PreFilter"), nodeNames:      []string{"node1"}, pod:            st.MakePod().Name("p").UID("p").Priority(highPriority).Req(largeRes).Obj(), pods: []*v1.Pod{ st.MakePod().Name("p1").UID("p1").Node("node1").Priority(midPriority).Req(smallRes).StartTime(epochTime).Obj(), }, expected: []string{"node1"}, }, { </del> name:           "a pod that fits on both nodes when lower priority pods are preempted", registerPlugin: st.RegisterPluginAsExtensions(noderesources.FitName, noderesources.NewFit, "Filter", "PreFilter"), nodeNames:      []string{"node1", "node2"},

Related discussion:
text: Other than that, I think we need to clearly document what the preemption logic does (that it executes the prefilter extensions points and the filter plugins multiple times) and the expectations from filter plugins (that they may get executed more than once in the same cycle)
Is it a predicate extender or preemption extender? If it's a predicate extender, I don't quite think we found the root cause - b/c predicate extenders are not invoked during preemption (). It makes more sense if it's a preemption extender - if the preemption extender mutates the to make it invalid. If it's the case, could you verify it by adding debug info before and after L157 to check the validness of ?
That's always true :)
filed a PR to mitigate it.
It's a predicate extender. I'll debug it more.
Added some debug info. after and . Here's the result. returned 4 candidates, but one of candidates () has zero pod. The candidate happened to be the best candidate and hence caused the "" error. It should have nothing to do with since our extender doesn't support Preemption. Disabling it may change pod placements/candidates selection and bypassed the error. It's like a coincidence. To remove the panic, a simple fix is to check if is empty. If it is, do not add the node to candidates. Alternatively, a change to may work too. Here is the log with debug info.
This looks more promising. Back to digging into the root cause, it seems returns an empty but with . So that implies firstly we have a non-nil , and then in , the is separated into two list: violatingVictims and nonViolatingVictims. Finally the two victims list are tried one by one to run . It looks like returns {fits=true, nil error} for every victim, which is quite abnormal. I think it's problematic that every pod is retrievable and in which we can debug into.
Yes, should not return fit=true with empty pods in the first place, but it's hard to locate the problematic code in this function.
The following code in looks questionable to me. If is empty, should the function return nil, nil? Also, if is true for all extenders, the loop won't do anything and candidates will remain the same.