jiebi/CodeConvo huggingface dataset:
Code change:
<del> apiVersion: extensions/v1beta1 </del> <ins> apiVersion: networking.k8s.io/v1beta1 </ins> kind: Ingress metadata: name: hostname spec: backend: serviceName: hostname <del> servicePort: 80 </del>  No newline at end of file <ins> servicePort: 80 </ins>

Related discussion:
text: <!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: Which test(s) are failing: Since when has it been failing: Testgrid link: https://k8s- Reason for failure: Anything else we need to know: /cc /milestone v1.19 /priority important-soon /sig network
Ref:
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry. ‍
my fu points me to this commit [] Fix multiple conversion tests that belongs to this PR , that seems related can you confirm if that PR is causing this test to fail? (should we gate on this test?)
Yes, that PR is related to the failure. The test utility is trying to convert extensions/v1beta1 data read from various files directly to a go type. This was never a valid thing to do, and happened to silently succeed because the two structs had an identical structure. This is exactly the type of bug was designed to flush out
the correct fix is to update the test/e2e/testing- files to
/assign
It was my PR that broke this - I can also fix that.
Thanks, o it was the commit that was good but the test that was wrong :upsidedownface: , At least I got right the commit :sweat_smile:
is out for review (talked with offline, he was aware that I'll send that out)
Code change:
for k, v := range pod.Annotations { copyPod.Annotations[k] = v } <del> copyPod.Annotations[kubetypes.ConfigMirrorAnnotationKey] = kubetypes.MirrorType </del> <ins> copyPod.Annotations[kubetypes.ConfigMirrorAnnotationKey] = getPodHash(pod) </ins> _, err := mc.apiserverClient.Pods(copyPod.Namespace).Create(&copyPod) return err

Related discussion:
text: See for more context. Currently, kubelet compares the spec of a mirror pod (created via apiserver) with that of a static pod (config stored as a file) to decide whether the two are considered identical. If they differ, kubelet would delete the mirror pod and re-create it with the correct spec. Due to the fact that apiserver may set default values for some fields in the spec of the mirror pod, the specs could deviate undesirably. To avoid problems like this, we can store the config of a static pod as an annotation in the mirror pod, and compare the static pod with the annotation.
Would a hash of the config be sufficient if we're only going to compare? The whole spec would generally be more useful, but I wonder if that is too much data to put in an annotation.
This seems totally reasonable to me, but I'm still desperately curious why you can't just bypass defaulting when you generate mirror pod from static pod? (i.e. just use the static pod's configuration verbatim)
It's not kubelet who applies the default values here. The admission plugin was only run at the apiserver, and to bypass that, you have to ask apiserver to treat mirror pods differently and potentially at multiple places.
On May 22, 2015 10:38 AM, "Yu-Ju Hong" wrote: Understood, but is it really multiple places? If so then I agree it's a bad idea. Otherwise it might be worth considering as an alternative.
It's untenable for Kubelet to assume to know/circumvent all fields that will be set by other entities: admission control, defaulting, late initialization, auto-something, whatever. It needs to obey the same rules as other clients.
I don't understand why you would ever want to apply any of the things you mentioned to static pods. Static pods should be copied verbatim into the mirror pods, right? That's what I'm proposing.
What if the master is updated before the kubelet, and the master adds a new API field with a default value?
That case should be discussed in
I agree that mirror pod should reflect the true status of the static pod running on the node, and no default values should be set in this case.
Code change:
for k, v := range pod.Annotations { copyPod.Annotations[k] = v } <del> copyPod.Annotations[kubetypes.ConfigMirrorAnnotationKey] = kubetypes.MirrorType </del> <ins> copyPod.Annotations[kubetypes.ConfigMirrorAnnotationKey] = getPodHash(pod) </ins> _, err := mc.apiserverClient.Pods(copyPod.Namespace).Create(&copyPod) return err

Related discussion:
text: However, I think it'd be harder to implement this way since, as stated above, that there are many things to circumvent in the apiserver itself. It could also be harder to keep this special case from being broken by a yet another new feature/plugin. Keeping the hash of the config in the annotation seems like a good compromise even though it may not reflect the true status 100% accurately. I think this is acceptable. What do you think?
Sure, if you think the annotation approach is easier, I would go with that. I guess in the case Brian mentoined (master adds a new API field with a default value), you would not update the mirror pod, because the static pod definition has not changed?
Yes, a hash would be sufficient.
It shouldn't be necessary to update the mirror pod in the case that a new default field was updated, because that field should be populated automatically by the apiserver on decode from etcd.
We should fix this so we don't have this problem again in 1.2.
Code change:
node, victims, nominatedPodsToClear, err := sched.config.Algorithm.Preempt(preemptor, sched.config.NodeLister, scheduleErr) if err != nil { <del> klog.Errorf("Error preempting victims to make room for %v/%v.", preemptor.Namespace, preemptor.Name) </del> <ins> klog.Errorf("Error preempting victims to make room for %v/%v: %v", preemptor.Namespace, preemptor.Name, err) </ins> return "", err } var nodeName = ""

Related discussion:
text: <!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Default scheduler crash if scheduler extender filter returns a not found node What you expected to happen: Default scheduler just should consider the not found node as not an candidate, instead of crash itself (panic). How to reproduce it (as minimally and precisely as possible): cluster only have 1 node: 0.0.0.1 scheduler extender (with nodeCacheCapable enabled) filter accidentally returns a candidate node: 0.0.0.2 scheduler recieved the candidate nodes: 0.0.0.2 from scheduler extender, but got below error: The is SuggestedHost: filteredNodes[0].Name: Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig scheduling
Code change:
"description": "ServicePort contains information on service's port.", "properties": { "appProtocol": { <del> "description": "The application protocol for this port. This field follows standard Kubernetes label syntax. Un-prefixed names are reserved for IANA standard service names (as per RFC-6335 and http://www.iana.org/assignments/service-names). Non-standard protocols should use prefixed names such as mycompany.com/my-custom-protocol. This is a beta field that is guarded by the ServiceAppProtocol feature gate and enabled by default.", </del> <ins> "description": "The application protocol for this port. This field follows standard Kubernetes label syntax. Un-prefixed names are reserved for IANA standard service names (as per RFC-6335 and https://www.iana.org/assignments/service-names). Non-standard protocols should use prefixed names such as mycompany.com/my-custom-protocol. This is a beta field that is guarded by the ServiceAppProtocol feature gate and enabled by default.", </ins> "type": "string" }, "name": {

Related discussion:
text: We should make the change in PR API reference links to should use HTTPS, not HTTP Visit h and look at the IANA hyperlinks Once this fix is made, a PR to is needed to update the docs (by autogenerating them). This typically would be deferred until the next Kubernetes release. v1.23 not applicable not applicable not applicable not applicable not applicable
/sig docs
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I opened this issue so that someone else can open a new PR that is equivalent to .
/retitle Update links to IANA for Service Name and Transport Protocol Port Number Registry
I would like to work on this issue. May I assign it to myself?
Anyone - including you - is welcome to work on this issue! So long as you've you can open a pull request to propose a fix.
Thanks
/assign
I have made the PR. I hope this works. If not then will be happy to do the necessary changes.
Had to close the previous PR because of issues, here is the new one I made.
Code change:
// Read retrieves data with the given "key" from CycleState. If the key is not // present an error is returned. <del> // This function is not thread safe. In multi-threaded code, lock should be // acquired first. </del> <ins> // This function is thread safe by acquiring an internal lock first. </ins> func (c *CycleState) Read(key StateKey) (StateData, error) { <ins> c.mx.RLock() defer c.mx.RUnlock() </ins> if v, ok := c.storage[key]; ok { return v, nil }

Related discussion:
text: What happened: kube-scheduler () crashes due to a panic while running : The race condition is triggered when an out-of-tree plugin, that writes to during its Filter phase, is registered through . Access to in general happens from a single goroutine and there are no practical concurrency issues, with the exception of the phase, that parallelizes calls for each node across plugins: A quick glance at the tip of the branch seems to indicate the bug is still present there. What you expected to happen: Ideally access to should be synchronized everywhere for correctness, including by all in-tree plugins, since there are no guarantees about how will call multiple plugins concurrently. It seems to me it is something that was considered at some point, given that has methods. Sadly, almost no in-tree plugins are synchronizing access. For now, I will try to work around the problem by doing a single write to during and storing a data bag structure where I will write state into during (with proper synchronization), instead of writing to directly. How to reproduce it (as minimally and precisely as possible): Install a scheduler plugin that writes to during its phase. In-tree (builtin) plugins are not synchronizing their access to . Environment: Kubernetes version (use ): v1.18.8 Cloud provider or hardware configuration: linux-amd64 (AWS) OS (e.g: ): Ubuntu 18.04.5 LTS (Bionic Beaver) Others: custom kube-scheduler binary built with an out-of-tree plugin using
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling
AFAICT all in-tree plugins write to the cycle state in PreFilter and then only read from cycle state in Filter. So this shouldn't be a problem with in-tree plugins as concurrent read access should be fine during RunFilterPlugins. But since your out-of-tree plugin writes in Filter, I can see why this can be a problem.
Code change:
// Read retrieves data with the given "key" from CycleState. If the key is not // present an error is returned. <del> // This function is not thread safe. In multi-threaded code, lock should be // acquired first. </del> <ins> // This function is thread safe by acquiring an internal lock first. </ins> func (c *CycleState) Read(key StateKey) (StateData, error) { <ins> c.mx.RLock() defer c.mx.RUnlock() </ins> if v, ok := c.storage[key]; ok { return v, nil }

Related discussion:
text: Aside from adding Lock/Unlock in in-tree plugin, offering mutual exclusion on each key as a built-in feature within CycleState could be an alternative solution. This way, reads from in-tree plugins during Filter wouldn't interfere with writes during Filter from out-of-tree plugins.
cc
should we change the Read/Write CycleState methods to be thread safe?
Adding locks without a proper justification would just increase the latency. could you clarify your use case? Why do you need to write anything during Filter?
yeah, we should measure it, but I don't expect it to have significant effect...
Adding locks without a proper justification would just increase the latency. Without contention (which is the case with in-tree plugins), I don't think this is a concern.
Adapting source from : source: essentially, 11.3 ns/Lock-Unlock pair
I remember locks showing up in profiling for PreScore/Score for certain plugins. Although I have to admit that the accesses were not properly implemented. I'd like to hear an answer for before doing any experiments.
+1 to that I'd also like to understand the motivations / requirements of mutating CycleState in Filter().
My particular use-case is that we have resources being dynamically to/removed from Nodes, and those resources need to be allocated (and tracked) in the scheduling loop. Technically I could pre-compute all the available dynamic resources for all Nodes at the phase, and snapshot it, but that would require me to pre-compute everything for all Nodes on all scheduling cycles. Instead, what I'm currently doing is computing what dynamic resources can be utilized on-demand during the phase (kind of a lazy loading of dynamic resources for each pair), and saving state of what extra resources were considered during for each Node that was evaluated. Then, later in the scheduler plugin can lock-in the exact dynamic resources for the particular Node that was selected. I understand the concern with adding extra synchronization overhead without a real need. Even if my use-case is not good enough to justify it, there is another argument for it: technically, the lack of access synchronization on violates the . Even if in the end we decide synchronizing all access is not worth the overhead (for correctness), I would strongly encourage better documentation saying which plugin phases are allowed to modify , and perhaps even forcing a snapshot-read-only copy of to be used by plugins.
Using the scheduler to track resources is not a good idea.
Code change:
// Read retrieves data with the given "key" from CycleState. If the key is not // present an error is returned. <del> // This function is not thread safe. In multi-threaded code, lock should be // acquired first. </del> <ins> // This function is thread safe by acquiring an internal lock first. </ins> func (c *CycleState) Read(key StateKey) (StateData, error) { <ins> c.mx.RLock() defer c.mx.RUnlock() </ins> if v, ok := c.storage[key]; ok { return v, nil }

Related discussion:
text: The scheduler should be stateless. Can't you track the resources with an external controller or with a device plugin and making it part of the Node spec?
there may be some misunderstanding in my poor choice of words. What I mean by "tracking" is that the scheduler needs to track usage and availability of these extra per-Node resources, to be able to allocate them during placement/scheduling. Exactly the same it does with volumes, cpus, memory, ephemeral disk, etc Doing computations in a controller outside of the scheduling loop would be similar to "precomputing" these resources. In my particular use case I chose a different way because: introduces another point of failure, if the external controller stops pre-computing extra resources and publishing them somewhere (CRD, Node spec, etc) scheduling halts (at least scheduling on these extra resources) Node resources in my use case are volatile (they are coming and going frequently), and the extra latency by an extra controller (vs doing all computations right in the scheduling loop) may be problematic for my use case. Even if you consider my use case not enough to add proper synchronization to , I'd still encourage making it explicit what scheduling phases are allowed to modify it (and even passing in a snapshot read-only version of to phases that are not expected to update it), to avoid confusion and race conditions.
Ok, if you are not tracking the existence of resources in the scheduler, I wouldn't suggest the external controller. Still, have you considered the device plugin framework? That way, the default kube-scheduler could keep track of resources usage. Otherwise, I still don't see why you would need to update the CycleState during Filter. As its name indicates, CycleState only lives during a scheduling cycle.
Regardless of the correctness of usage of CycleState to do what wants, I still think we need to do at least one of the following: that writing to CycleState during Filter will lead to data races, a read-only CycleState to Filter plugins (whether we do this at the interface level or if we simply drop Write requests during Filter is an implementation detail), or make CycleState concurrency safe CycleState as it exists right now allows plugin authors to shoot themselves in the foot IMO.
I'm leaning towards 1, but it's fair to continue the discussion before deciding.
Can you run the "controller" as a goroutine along with your scheduler plugin?
Code change:
// Read retrieves data with the given "key" from CycleState. If the key is not // present an error is returned. <del> // This function is not thread safe. In multi-threaded code, lock should be // acquired first. </del> <ins> // This function is thread safe by acquiring an internal lock first. </ins> func (c *CycleState) Read(key StateKey) (StateData, error) { <ins> c.mx.RLock() defer c.mx.RUnlock() </ins> if v, ok := c.storage[key]; ok { return v, nil }

Related discussion:
text: So it's not an external controller binary, and hence don't introduce another point of failure. It's pretty like how the default scheduler manages its internal cache. But note that cache is only needed when you need a more efficient data structure. If a {key:value} map is good enough, spinning up a XYZInformer and obtain the latest XYZ resource on-demand is the most recommended way. I think it quite depends on how you design the internal XYZ cache. For example, it should only tweak the cache in an incremental manner - in other words, unchanged resource will be kept unmodified. You can check out the "doubly-pointer snapshot" of scheduler's cache. With above said, if I understand your case correctly, you may need to build a sophisticated XYZ cache, which runs in a separate goroutine and responds to your XYZ resource changes. And then upon each scheduling cycle, take a snapshot (may needs a specific snapshotting algorithm) in PreFilter() and WRITE to CycleState, and then in Filter() READ from CycleState. My personal option is to suggest all plugin developers to read from CycleState in Filter().
nailed it. What you describe is almost exactly what I’m doing today. There are some details particular to my use case (e.g. just tracking current state with an informer and an incremental cache is not enough, because there’s some logic involved in deciding if currently available resources are usable by each pair, and I currently do that computation in as pairs are evaluated). In any case, I just wanted to share my use case to provide some color, but I don’t want what I’m doing distracting us from the more important questions, mentioned by above. My personal preference would be as a very short term next step, then when someone has cycles to do it. I would be OK with punting on until a more strong need shows up.
Note that limiting to only reading from works for me, as long as I can still read my own data structure from it and do writes on it (i.e. don’t deepclone for the phase).
also adds overhead (from copying the state, which can be very expensive).
Could implement just by changing the interface passed to (e.g. ). That wouldn’t be backwards compatible, but very cheap.
Code change:
// Read retrieves data with the given "key" from CycleState. If the key is not // present an error is returned. <del> // This function is not thread safe. In multi-threaded code, lock should be // acquired first. </del> <ins> // This function is thread safe by acquiring an internal lock first. </ins> func (c *CycleState) Read(key StateKey) (StateData, error) { <ins> c.mx.RLock() defer c.mx.RUnlock() </ins> if v, ok := c.storage[key]; ok { return v, nil }

Related discussion:
text: I guess that would work in the sense that you cannot change the high level values of the CycleState map, but you could still override what's inside them, if they have pointers. But maybe that's ok.
but you could still override what's inside them, if they have pointers. But maybe that's ok. yeah, if it was pointers and if the underlying data needed to be modified, I'd imagine it would have its own Lock/Unlock semantics
for such implementation, the only remaining concern is backwards compatibility. So I would leave that decision to who leads the scheduling-plugins repo.
I rethink 's original problem a little bit: the original racing issue occurs when reading CycleState in in-tree plugins, this indicates one of the (out-of-tree) plugins is (over)writing the most high-level value of the CycleState map. But if we control the locking inside the out-of-tree plugin, and mutate that particular in-place, that would be good enough? Case 3 in the above program demonstrates the correct way to manipulate the specific out-of-tree state. Let me know your thoughts. If it resolves 's concern, we don't need to make any change on the Filter() interface.
Case 3 is how I am working around the problem today to avoid the race. I write a "data bag" once into top level in , then only modify my own inner structure (with proper locking) in , without doing any further modifications to directly (top level) in . That is acceptable, but I'd encourage you to consider making the interface (and documentation) more explicit for plugins, so they avoid the mistake of writing directly into . Without changing anything (no documentation, no readonly interface), it is pretty easy for a plugin to use in a way that you don't want to support.
Thanks for confirming that. Yes, documenting it, or(and) publishing examples would be quite helpful to clarify the semantics. As a wrap-up: a keyed value can be read from CycleState, literally , in any phase. It's always race-free.