jiebi/CodeConvo huggingface dataset:
Code change:
SwaggerProps: spec.SwaggerProps{ Paths: &spec.Paths{ Paths: map[string]spec.PathItem{ <del> "/apis/apiservicegroup/v1": {}, </del> <ins> "/apis/apiservicegroup/v1/path1": {}, </ins> }, }, },

Related discussion:
text: Can I get your exact repro steps? suggests that the conflicting paths should be removed before the merge. We always call this function with
One other question is if this is reproducible by just applying a set of yaml/installations or does it require a cluster upgrade to trigger? Thanks!
hit same problem and we cannot install any other new helm plugins. maybe we need to move forward
Here are verbose repro instructions. An apiserver restart on 1.28 will induce the issue.
Hello. I'm still having some trouble reproducing although I'm not 100% sure if the configuration is correct. I'm trying to repro via kind. afaik installing via helm has prometheus adapter serving . seems to suggest that prometheus adapter is serving under . Applying the update to the configmap per the instructions didn't change the apiservice on my end. Not sure if this is my misconfiguration or intended. I've also tested via integration test that having two openapis serving the same path doesn't cause errors and one will get filtered out. A simple workaround is probably to delete the metrics-server if it is already superseded by the prometheus-adapter.
The is not set up in an resource for us, we only created an APIService with . When I go fetch the openAPI schema served up by the endpoint, it has schema information for in it if that is enabled using . If I remove then the schema fetched from the server doesn't have stuff in it, and the error is resolved. We have been keeping around because I think something uses it directly, or used to. Maybe ? That could have changed But in any case, this problem is quite severe when it happens so it would be nice to fix it as there may be others who do the same update and have to do some deep diving into the logs and all kinds of crazy diagnostics and guesswork to figure it out.
How does the get reached without an APIService? The resources would be unavailable for the apiserver to query and it'll be through something else? I see, adding resources for me doesn't populate the OpenAPI. I wonder if this could be cloud provider specific. I'm totally supportive of fixing, it's just quite difficult to reproduce on my part .
Hmm good question.
Code change:
SwaggerProps: spec.SwaggerProps{ Paths: &spec.Paths{ Paths: map[string]spec.PathItem{ <del> "/apis/apiservicegroup/v1": {}, </del> <ins> "/apis/apiservicegroup/v1/path1": {}, </ins> }, }, },

Related discussion:
text: In our staging cluster I tried bringing back the in the custom metrics server configuration and restarting all the masters, and the problem did not come back, everything seems OK now. So it maybe isn't that easy to reproduce
After updating the configmap, restarting the prometheus-adapter pod to pickup the changes and restarting the apiserver are both required to induce the issue. I have been able to deterministically reproduce this following those steps, please follow up here if you are still having issues.
That's good to hear. I'll need some more help with reproducing it. I've been able to verify that restarting the prometheus adapter pod picks up the configmap changes but restarting the apiserver hasn't caused any crashes for me. I've tested this both on the latest master branch and 1.28.
While I'm yet to fully understand these changes, the 2 PRs below (both of which went into 1.28) seem likely candidates for a regression here: - introduced lazy loading in openAPI v2 handler to load/compute the aggregator spec when requested by client - switched the spec merging function from to any clue if one of the above changes could be causing mischief?
Also from example request at the top, the impact seems to be to the v2 API: Nathan - when you get a chance, could you also confirm if v1 API doesn't run into this?
OpenAPI only supports v2 and v3.
hi, any clue about this issue?
It's possible but I haven't been able to reproduce it. Can I ask if this is happening on cloud providers outside AWS?
Here are the apiservices and relevant pod objects with a working repro from these steps If it's useful, I can provide yamls for these objects in addition to the settings applied in the prometheus-adapter configmap. Can you confirm that the pods backing your prometheus-adapter and metrics-server are running?
I meet this problem
We the problem in GCP as well.
ACK in Alibaba Cloud also meets this problem
We were able to fix the v2 availability by and re-installing the kube-prometheus helm chart. Unfortunately, after restarting the k8s api server, v2 availability is broken again.
yes, I'm running on kind. I'll do some more digging, assuming it's related to those PRs linked by but still haven't pinpointed anything.
Code change:
SwaggerProps: spec.SwaggerProps{ Paths: &spec.Paths{ Paths: map[string]spec.PathItem{ <del> "/apis/apiservicegroup/v1": {}, </del> <ins> "/apis/apiservicegroup/v1/path1": {}, </ins> }, }, },

Related discussion:
text: I tried to reproduce on my local setup with based on steps + , but I could still get a successful response when I executed . Then I reverted one of the step mentioned in to and waited until the new pod came alive and then did a apiserver restart and I was able to reproduce the same thing on as well i.e; I get the 503 response when I execute following command after the fact. After this point, even if i change it back to and restart the apiserver , the error is persistent. Let me know if this works for you to reproduce ? PS: Also, i want to call out that there is no issue with , its .
Thanks, alright I found a way to reproduce it consistently without restarting the apiserver. Still investigating the root cause. Thanks for the detailed reproduction and
Quick update: This issue was a combination of bugs both in k/k and prometheus adapter. I opened to mitigate on k/k side which should fix the issue. The issue was difficult to reproduce because it requires a very specific ordering of OpenAPI merging to trigger, which is why I suspect many people were able to reproduce after some restarts to kube-apiserver. Prometheus and metrics server serve differing OpenAPIs for the endpoint. Prometheus also doesn't seem to be registering discovery's OpenAPI properly (missing OpenAPI for but having the OpenAPI for whereas it is all present in metrics server. This inconsistency in how prometheus and metrics server handle the OpenAPI surfaced the merging issue.
a little wired question: why do we need to restart kube-apiserver to reproduce this problem?
Tried to reproduce the issue after cherry picking the fix and didn't run into below error. Fix is working as expected.
It can look like restarting prometheus-adapter "fixes" the issue? I am also curious about suggestion of removing the section from the config map works as a temporary workaround until released upstream (we use and cannot delete the apiregistration)? As I understand the fix for this issue is currently in
Code change:
<ins> vagrant: user.present: - optional_groups: - docker - remove_groups: False - require: - pkg: docker-io </ins>

Related discussion:
text: As a sysadmin When using the Vagrant cluster provided And logged in as the vagrant user to the master or minions I would like to be able to call docker commands wihout requiring the 'sudo' prefix. For Fedora =19 and RHEL=7 this can be enabled by adding the vagrant user to the docker group in /etc/groups
This should be trivial, but its tricker to do because of the whole salt configuration process as the docker group is only created on each kubernetes-minion once docker is installed by the salt-minion. To do this properly, we would need a salt state change for this, and new flag in to let you specify additional users that should be in the docker group.
Code change:
go test -c  ${goflags:+"${goflags[@]}"}  -gcflags="${gogcflags}"  <del> -asmflags="${goasmflags}"  </del> -ldflags="${goldflags}"  -tags="${gotags:-}"  -o "${outfile}" 

Related discussion:
text: While working on for , I found that our kubernetes1.24 package stopped to produce bit-identical results on 2 fresh builds. The first chunk is likely just the hash of the remainder, so can be ignored. The other chunks indicate that a random build temporary path is used for Sources should be able to reproduce bit-identical binaries e.g. for SLSA level4 On Debian or openSUSE do No response <detailsWARNING: This version information is deprecated and will be replaced with the output from kubectl version --short. Use --output=yaml|json to get the full version. Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2022-05-05T00:00:00Z", GoVersion:"go1.18.3", Compiler:"gc", Platform:"linux/amd64"} Kustomize Version: v4.5.4 console $ cat /etc/os-release $ uname -a Linux bmwiedemann-factory 5.18.6-1-default SMP PREEMPTDYNAMIC Thu Jun 23 05:46:18 UTC 2022 () x8664 x8664 x86_64 GNU/Linux </details<detailsgo1.18 </details<details</details<details</details
/sig release /area build-release
can you please post a link to the full logs from the build?
any news on this? Haven't heard from (context: we worked to fix this a couple of times in )
Unfortunately not, is this something we can fix on the OBS side? If the go build timestamp () is the main issue here then we can approach a fix in our build pipeline.
It would be good if would not be embedded in the first place. It might be possible to replace it somehow in our builds, but all other people wanting reproducible builds, would need to replicate that.
full logs from the build: from kubernetes-1.24.3 built 2022-08-18
can you please try setting GOCACHE=off when you run the build? I am suspecting that the temp directory is being generated here: (or add an export in for that env var)
I to but still got these diffs.
Code change:
go test -c  ${goflags:+"${goflags[@]}"}  -gcflags="${gogcflags}"  <del> -asmflags="${goasmflags}"  </del> -ldflags="${goldflags}"  -tags="${gotags:-}"  -o "${outfile}" 

Related discussion:
text: So i built golang from source, peppered it with changes to track down where this is getting created and found that it is coming from: I don't know yet how to fix it. will dig more when i can (travelling for the next few days)
Would it work to override the , I see that they use the : Ref: https://reproducible- Edit: Probably not, we may make it configurable in golang :thinking:
this is so tricky :( will dig more when i get back from my event this week
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close not-planned [1]:
Closing this issue, marking it as "Not Planned".
Code change:
go test -c  ${goflags:+"${goflags[@]}"}  -gcflags="${gogcflags}"  <del> -asmflags="${goasmflags}"  </del> -ldflags="${goldflags}"  -tags="${gotags:-}"  -o "${outfile}" 

Related discussion:
text: <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This is still relevant even with 1.27. It might be worth revisiting this issue again. /reopen /triage accepted
Reopened this issue. <detailsIn response to : This is still relevant even with 1.27. It might be worth revisiting this issue again. /reopen /triage accepted Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle rotten
latest run -
I found that our package has the same issue, so it probably it is a toolchain issue in golang - and much faster to debug outside of the slow kubernetes build. =filed
AIUI, the conclusion by the golang people seems to be that commit is not the right approach and one should use with the go call instead. Not sure how that interacts with other parts of your build.
cc (please see referenced commit above!)
I see. I inherited the gcflags before so I plead innocence. It seems plausible to change it from to - I don't know when I will have time to look at that, though.
Code change:
go test -c  ${goflags:+"${goflags[@]}"}  -gcflags="${gogcflags}"  <del> -asmflags="${goasmflags}"  </del> -ldflags="${goldflags}"  -tags="${gotags:-}"  -o "${outfile}" 

Related discussion:
text: My patch doesn't seem to fix the whole problem: thockin-glaptop4 kubernetes buildtrimpath /$ make kubelet +++ [1114 19:10:12] Building go targets for linux/amd64 (non-static) thockin-glaptop4 kubernetes buildtrimpath /$ md5sum output/bin/kubelet output/bin/kubelet thockin-glaptop4 kubernetes buildtrimpath /$ make kubelet +++ [1114 19:12:40] Building go targets for linux/amd64 (non-static) thockin-glaptop4 kubernetes buildtrimpath /$ md5sum output/bin/kubelet output/bin/kubelet thockin-glaptop4 kubernetes build_trimpath /$
diff --- a 2023-11-14 19:13:45. -0800 +++ b 2023-11-14 19:13:56. -0800 -1,4 +1,4 -38rZHdQckYF9EEzmndR/BAXOwzCcq0Mk1UHT62ha/QLiRgXP4oojWh3oWP9/TWG7KW90IL8vKt1UOEPR +LeLArvl7sgWAd55ROKN2/BAXOwzCcq0Mk1UHT62ha/QLiRgXP4oojWh3oWP9/cYb5ZGQwa6js077YiAvK /lib64/ld-linux-x86- AUATUSH []AA] -,8 +,8 go1.21.3 runtime/cgo: call frame too large -2023-11-15T03:12:40Z -2023-11-15T03:12:40Z +2023-11-15T03:13:47Z +2023-11-15T03:13:47Z pthreadcreate failed: %s bad stack bounds: lo=%p hi=%p expand 32-byte kexpand 32-byte k
That seems unrelated. I tested with and it made our openSUSE package fully reproducible. Your diff contains a timestamp - maybe would get rid of that. The top part of the diff is just the go build-id hash of the remainder and can be ignored.
Code change:
go test -c  ${goflags:+"${goflags[@]}"}  -gcflags="${gogcflags}"  <del> -asmflags="${goasmflags}"  </del> -ldflags="${goldflags}"  -tags="${gotags:-}"  -o "${outfile}" 

Related discussion:
text: Thanks for confirming
Will there be backports to 1.24++ ? I found it applies cleanly to 1.26++ and works with some small tweaks in 1.24 and 1.25. If not, I'd submit the patch for our openSUSE packages.
1.24 and 1.25 reached EOL and we don't support these branches any longer. We can cherry-pick this to 1.26+ if folks ( agree
Cherry-picks: 1.29 1.28 1.27 1.26
Code change:
w.Write(LEVEL_0, "Concurrency Policy:t%sn", cronJob.Spec.ConcurrencyPolicy) w.Write(LEVEL_0, "Suspend:t%sn", printBoolPtr(cronJob.Spec.Suspend)) if cronJob.Spec.SuccessfulJobsHistoryLimit != nil { <del> w.Write(LEVEL_0, "Successful Job History Limit:t%dn", cronJob.Spec.SuccessfulJobsHistoryLimit) </del> <ins> w.Write(LEVEL_0, "Successful Job History Limit:t%dn", *cronJob.Spec.SuccessfulJobsHistoryLimit) </ins> } else { w.Write(LEVEL_0, "Successful Job History Limit:t<unset>n") }

Related discussion:
text: What happened: The history limit in the describe display has the incorrect number What you expected to happen: The history limit should show the default limit (3) How to reproduce it (as minimally and precisely as possible): Add a cronjob with default history limit settings and run Anything else we need to know?: Environment: Kubernetes version (use ): $ kubectl version Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-04-08T17:11:31Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-04-08T17:02:58Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: Brightbox OS (e.g: ): Ubuntu 18.04.2 Kernel (e.g. ): Linux bionic-vm 4.15.0-48-generic -Ubuntu SMP Wed Apr 3 08:28:49 UTC 2019 x8664 x8664 x86_64 GNU/Linux Install tools: kubeam Network plugin and version (if this is a network-related bug): Others:
/sig cli
/assign
Code change:
return w, nil }) <del> // Create second pod which should preempt first pod. </del> <ins> // Create third pod which should preempt other pods. </ins> preemptorPod, err := createPausePod(testCtx.ClientSet, <del> initPausePod(&pausePodConfig{Name: "preemptor-pod", Namespace: testCtx.NS.Name, Priority: &highPriority, Resources: &resourceRequest})) </del> <ins> initPausePod(&pausePodConfig{Name: "preemptor-pod", Namespace: testCtx.NS.Name, Priority: &highPriority, Resources: &preemptorResourceRequest})) </ins> if err != nil { t.Errorf("Error while creating the preemptor pod: %v", err) }

Related discussion:
text: <!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via -- We may delete podS though they may be waiting in Do not delete waiting pods if they are not scheduled. Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling
/cc WDYT ?
It sounds to me that you prefer to prioritize waiting Pod over other assigned Pods. In that case, you may want to exclude waiting Pods in the earlier Filtering/Sorting Victims step, instead of simply disregard the deletion in PrepareCandidate(). In general, when Permit plugins work along with PostFilter (preemption), the logic could be complicated. There may not be a general preemption implementation (probably in the future we can exercise pluginArgs to provide different options). So for now the best bet is to implement your own PostFilter plugin.
Got it! But for the default preemption plugin, we delete the victims to free resources. If pod has assigned, we should delete it. If pod has not assigned, we just need reject it in WaitingPod because the pod just exists in the cache.
Oh, your point was to verify if it's a waiting pod first, and then delete it? If so, then it makes sense to me. Feel free to raise a PR. /assign
Yes, I will open a PR to do this
Code change:
for _, item := range testCases { stopCh := make(chan struct{}) fakeClientset := fake.NewSimpleClientset(&v1.PodList{Items: item.pods}) <del> controller := NewNoExecuteTaintManager(fakeClientset, getPodFromClientset(fakeClientset), (&nodeHolder{item.newNode}).getNode, getPodsAssignedToNode(fakeClientset)) </del> <ins> controller := NewNoExecuteTaintManager(fakeClientset, getPodFromClientset(fakeClientset), (&nodeHolder{node: item.newNode}).getNode, getPodsAssignedToNode(fakeClientset)) </ins> controller.recorder = testutil.NewFakeRecorder() go controller.Run(stopCh) controller.NodeUpdated(item.oldNode, item.newNode)

Related discussion:
text: What happened: A Pod was scheduled for deletion because a taint w/ effect was observed in the Node the Pod was assigned to, and after the taint is removed from the Node, the Pod scheduled deletion isn't canceled. After scheduling pod deletion, the only cancels pod deletion if all taints are removed from the node, including any taints that the user relies on for their use-cases (example below). What you expected to happen: After the taint that triggered scheduling pod deletion is removed, cancels the scheduled pod deletion. How to reproduce it (as minimally and precisely as possible): A node is tainted w/ , so any workloads already assigned to it which don't tolerate this taint are evicted. A pod is created w/ tolerations: w/ unspecified and w/ . is assigned to . At some point in time after the previous steps took place, is tainted as . gets an update event for and observes two taints. It proceeds to calculate the minimum time (in seconds) that tolerates for the two taints which at this point is 300 seconds, and marks for deletion in ~300 seconds - this happens in-memory, in a timed-queue. At some point in time after the previous steps took place, and before the 300 seconds trigger happen, taint is removed from . gets an update event for and observes only one taint, . It proceeds to calculate the minimum time (in seconds) that tolerates for the this taint which is infinity, and returns, not canceling the previous deletion. It completely ignores the fact that the taint that triggered deletion is no longer observed. Anything else we need to know?: Worked w/ to figure this out after all pods in a node got evicted without apparent reason. Environment: Kubernetes version (use ): 1.17.3 (but looking at the code seems to affect all of 1.17.x and 1.18.x and master. Cloud provider or hardware configuration: N/A OS (e.g: ): N/A Kernel (e.g. ): 5.x Install tools: N/A Network plugin and version (if this is a network-related bug): N/A Others: N/A cc (owners of this code)
There seems to be at least .