jiebi/CodeConvo huggingface dataset:
The "Kubernetes-y" thing to do here would be to add a status component to the 3rd party resource object and have each apiserver report when they've applied the change, then you'd wait for that instead of your current 404-loop. Having the POST method itself do the waiting is going to be a really tough sell. Watch is intended to handle all corner cases involved when clients are waiting on information from the server, POST is not equipped to do that. (e.g., what happens if the apiserver you POSTed to crashes while waiting? You wouldn't know if you should re-post or not and you wouldn't have any way to figure out the status at that point.) , Doug Davis wrote:
I wouldn't argue for a single-master exception, nor would I argue for a POST that waits. I am unfamiliar with how everything works, so I am having a little trouble parsing some of these responses. I see mentioned a and . What are those? Are they applicable here? Is there a way to watch for this change to be made that is built in, or is the way to go for now?
Why was this closed? is a doc issue, I don't think this is a doc problem.
I am not so concerned about the UX issue if the delay really is isolated to system start-up (as if there is such a thing). What about this scenario: operator registers a TPR, then waits for all the apiservers running at the time to fully react, and thereafter nothing is prepared for the TPR to be absent. Some time later another apiserver is , or an existing one crashes and is re-started. Will there be an interim during which the new(ly started) apiserver is not ready for clients to use the TPR?
would know better, but I'd expect the readiness check to fail until the initial sync of registered apiservice objects completed, preventing the new API server from making it into the LB pool
will report unhealthy until the apiservices are synced, but TPR was based on an unusual polling technique that can take about 10 seconds. I'm not certain when it will be ready. CRD is based on the state of an informer's list/watch, so its much more current.
This consistently works well for CRDs. modified script to use CRDs:
Closing since this works for CRDs. /close
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: What you expected to happen: I expected the update to run successfully. How to reproduce it (as minimally and precisely as possible): Checkout. . Anything else we need to know?: I get this in both master and release-1.9. I tested most carefully in release-1.9 (because I am working on a cherry-pick right now). In particular, I blew away the container and image I had used earlier for building. Before running I established this state: Then I ran --- which took a while but eventually completed. It left me this state: After that is when I ran the that is exhibited at the start of this issue. Environment: Kubernetes version (use ): Commit (current release-1.9) Cloud provider or hardware configuration: local VM OS (e.g. from /etc/os-release): Ubuntu 17.10 Kernel (e.g. ): Linux ubu17dev 4.13.0-21-generic -Ubuntu SMP Mon Dec 18 17:29:16 UTC 2017 x8664 x8664 x86_64 GNU/Linux Install tools: Others: Docker 17.12.0-ce
Ah, there's another bug for this, this is being fixed. (same issue, verify uses update internally)
Any workaround?
/reopen It looks like the 1.8 and 1.9 release branches are broken for the same reason, but the fix is a bit less trivial, since we can't just use the latest gazelle, since this would require a huge mass of changes to the BUILD files. (Luckily, 1.7 doesn't use gazelle, so it's unaffected.) Some additional background: gazelle doesn't vendor bazelbuild/buildtools, though it does have a bazel dependency on a particular version. As a result, is always grabbing the latest version of bazelbuild/buildtools, even with our hacks to check out an older version of gazelle. An additional complication for the 1.8 and 1.9 branches is that we're installing gazelle from bazelbuild/rulesgo instead of bazelbuild/bazel-gazelle.
Various options under consideration to fix these release branches: vendor gazelle, similar to (but not a straight cherrypick, since versions are different) the bazelbuild/rulesgo repo somewhere (where?) and then explicitly vendor and older version of bazelbuild/buildtools, and then use that fork instead of the upstream fork the hack in to also check out a particular bazelbuild/buildtools revision (so hacky) Anyone have a better idea?
you can't re-open an issue/PR unless you authored it or you are assigned to it. <detailsIn response to : /reopen Fully vendor gazelle, similar to (but not a straight cherrypick, since versions are different) Clone the bazelbuild/rulesgo repo somewhere (where?) and then explicitly vendor and older version of bazelbuild/buildtools, and then use that fork instead of the upstream fork Extend the hack in to also check out a particular bazelbuild/buildtools revision (so hacky) Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign /reopen
you can't re-open an issue/PR unless you authored it or you are assigned to it. <detailsIn response to : /assign /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I think we should start with 2) to get things fixed and then solve 1) for the current branch and come back to the release branches if we think we still need that.
can you describe 1) and 2) here?
I ended up implementing option 3 in ; it wasn't quite as gross as I thought it'd be. LMK what you think.
1) and 2) from the bullets in
ahh, thanks.
many thanks... would the same fix work in release-1.7 and release-1.8? trying to get those in shape as well
PRs are out for 1.8 and 1.9. The same thing would work for 1.7 AFAIK Edit: see below, I am wrong WRT 1.7
1.7 is unaffected because it doesn't use gazelle.
What happened: When we build with make to build k8s, it will use 'amd64' in the container names. This is not the case for bazel builds, which are still using something like 'kube-apiserver' instead of 'kube-apiserver-amd64' What you expected to happen: Images build with make or bazel should have 'amd64' in the container name How to reproduce it (as minimally and precisely as possible): make quick-release cat inside tar "RepoTags": ["" make bazel-release cat inside tar "RepoTags": [""
/sig cluster-lifecycle /assign
+1 we need to standardize both builds to generate the same. +1 to add to the bazel build
FYI
+1, cc
bazel / docker build parity SGTM.
+1. Perhaps we should document a standard, like: All docker images that are arch specific should have the arch in the name Arch is only able to be omitted when it's a manifest-based tag (multi-arch) Document legacy exceptions, and hopefully bring them into compliance /cc
i like it. where is the best spot to doc this?
-- please doc as updates to:
Created
When installing the metrics-server to implement the API and the prometheus-adapter to implement the API, it is possible to make the OpenAPI handler fail to initialize with this error log in apiserver: This will result in kubectl discovery calls to /openapi/v2 to fail with a 503: The OpenAPI handler should not fail to initialize if both and are serving the same path for different API groups/versions. I can provide exact installation steps if needed, however, a high-level repro is: install metrics-server to serve install prometheus-adapter to serve ensure resourceRules is populated in the prometheus-adapter config so that resource metrics are also served by restarting apiserver will result in the OpenAPI failure and 503 responses to /openapi/v2 path No response <details</details<detailsAWS </details<details</details<details</details<details</details<details</details
Please let me know if I should reopen the issue in
/sig api-machinery
/sig instrumentation [metrics apis]
Would you mind taking a look when have time? Thank you^^ /triage accepted
I have a question. should be serving under . Why does an apiservice serving another gvk register paths under ? cc we've made the assumption that an APIService (such as metrics-server or prometheus-adapter) serves only gvks under its own registered group version. I'll do some more investigation but just wondering if prometheus doesn't follow that assumption?
the prometheus-adapter can also be configured to serve and replace the metrics-server by enabling these metrics, However, if the prometheus-adapter fronts and metrics-server fronts , it does seem like the correct behavior would be for prometheus-adapter to serve until an operator updates the APIService to reference prometheus-adapter rather than metrics-server. We're seeing this issue occur for upgrades between 1.27 and 1.28 so it does look this path duplication did not cause the OpenAPI handler to fail on 1.27.
I just upgraded from 1.27 to 1.28 and running into this problem. Some kubectl apply operations fail, and ArgoCD cannot load its custom resources. Any suggested workaround? Would removing the from the prometheus configuration resolve it? Or perhaps disable metrics-server?
Simply looking at the code diff between 1.27 and 1.28 I can't pinpoint the cause.
Can I get your exact repro steps? suggests that the conflicting paths should be removed before the merge. We always call this function with
One other question is if this is reproducible by just applying a set of yaml/installations or does it require a cluster upgrade to trigger? Thanks!
hit same problem and we cannot install any other new helm plugins. maybe we need to move forward
Here are verbose repro instructions. An apiserver restart on 1.28 will induce the issue.
Hello. I'm still having some trouble reproducing although I'm not 100% sure if the configuration is correct. I'm trying to repro via kind. afaik installing via helm has prometheus adapter serving . seems to suggest that prometheus adapter is serving under . Applying the update to the configmap per the instructions didn't change the apiservice on my end. Not sure if this is my misconfiguration or intended. I've also tested via integration test that having two openapis serving the same path doesn't cause errors and one will get filtered out. A simple workaround is probably to delete the metrics-server if it is already superseded by the prometheus-adapter.
The is not set up in an resource for us, we only created an APIService with . When I go fetch the openAPI schema served up by the endpoint, it has schema information for in it if that is enabled using . If I remove then the schema fetched from the server doesn't have stuff in it, and the error is resolved. We have been keeping around because I think something uses it directly, or used to. Maybe ? That could have changed But in any case, this problem is quite severe when it happens so it would be nice to fix it as there may be others who do the same update and have to do some deep diving into the logs and all kinds of crazy diagnostics and guesswork to figure it out.
How does the get reached without an APIService? The resources would be unavailable for the apiserver to query and it'll be through something else? I see, adding resources for me doesn't populate the OpenAPI. I wonder if this could be cloud provider specific. I'm totally supportive of fixing, it's just quite difficult to reproduce on my part .
Hmm good question.
In our staging cluster I tried bringing back the in the custom metrics server configuration and restarting all the masters, and the problem did not come back, everything seems OK now. So it maybe isn't that easy to reproduce
After updating the configmap, restarting the prometheus-adapter pod to pickup the changes and restarting the apiserver are both required to induce the issue. I have been able to deterministically reproduce this following those steps, please follow up here if you are still having issues.
That's good to hear. I'll need some more help with reproducing it. I've been able to verify that restarting the prometheus adapter pod picks up the configmap changes but restarting the apiserver hasn't caused any crashes for me. I've tested this both on the latest master branch and 1.28.
While I'm yet to fully understand these changes, the 2 PRs below (both of which went into 1.28) seem likely candidates for a regression here: - introduced lazy loading in openAPI v2 handler to load/compute the aggregator spec when requested by client - switched the spec merging function from to any clue if one of the above changes could be causing mischief?
Also from example request at the top, the impact seems to be to the v2 API: Nathan - when you get a chance, could you also confirm if v1 API doesn't run into this?
OpenAPI only supports v2 and v3.
hi, any clue about this issue?
It's possible but I haven't been able to reproduce it. Can I ask if this is happening on cloud providers outside AWS?
Here are the apiservices and relevant pod objects with a working repro from these steps If it's useful, I can provide yamls for these objects in addition to the settings applied in the prometheus-adapter configmap. Can you confirm that the pods backing your prometheus-adapter and metrics-server are running?
I meet this problem
We the problem in GCP as well.
ACK in Alibaba Cloud also meets this problem
We were able to fix the v2 availability by and re-installing the kube-prometheus helm chart. Unfortunately, after restarting the k8s api server, v2 availability is broken again.
yes, I'm running on kind. I'll do some more digging, assuming it's related to those PRs linked by but still haven't pinpointed anything.
I tried to reproduce on my local setup with based on steps + , but I could still get a successful response when I executed . Then I reverted one of the step mentioned in to and waited until the new pod came alive and then did a apiserver restart and I was able to reproduce the same thing on as well i.e; I get the 503 response when I execute following command after the fact. After this point, even if i change it back to and restart the apiserver , the error is persistent. Let me know if this works for you to reproduce ? PS: Also, i want to call out that there is no issue with , its .
Thanks, alright I found a way to reproduce it consistently without restarting the apiserver. Still investigating the root cause. Thanks for the detailed reproduction and
Quick update: This issue was a combination of bugs both in k/k and prometheus adapter. I opened to mitigate on k/k side which should fix the issue. The issue was difficult to reproduce because it requires a very specific ordering of OpenAPI merging to trigger, which is why I suspect many people were able to reproduce after some restarts to kube-apiserver. Prometheus and metrics server serve differing OpenAPIs for the endpoint. Prometheus also doesn't seem to be registering discovery's OpenAPI properly (missing OpenAPI for but having the OpenAPI for whereas it is all present in metrics server. This inconsistency in how prometheus and metrics server handle the OpenAPI surfaced the merging issue.
a little wired question: why do we need to restart kube-apiserver to reproduce this problem?
Tried to reproduce the issue after cherry picking the fix and didn't run into below error. Fix is working as expected.
It can look like restarting prometheus-adapter "fixes" the issue? I am also curious about suggestion of removing the section from the config map works as a temporary workaround until released upstream (we use and cannot delete the apiregistration)? As I understand the fix for this issue is currently in
As a sysadmin When using the Vagrant cluster provided And logged in as the vagrant user to the master or minions I would like to be able to call docker commands wihout requiring the 'sudo' prefix. For Fedora =19 and RHEL=7 this can be enabled by adding the vagrant user to the docker group in /etc/groups
This should be trivial, but its tricker to do because of the whole salt configuration process as the docker group is only created on each kubernetes-minion once docker is installed by the salt-minion. To do this properly, we would need a salt state change for this, and new flag in to let you specify additional users that should be in the docker group.
While working on for , I found that our kubernetes1.24 package stopped to produce bit-identical results on 2 fresh builds. The first chunk is likely just the hash of the remainder, so can be ignored. The other chunks indicate that a random build temporary path is used for Sources should be able to reproduce bit-identical binaries e.g. for SLSA level4 On Debian or openSUSE do No response <detailsWARNING: This version information is deprecated and will be replaced with the output from kubectl version --short. Use --output=yaml|json to get the full version. Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2022-05-05T00:00:00Z", GoVersion:"go1.18.3", Compiler:"gc", Platform:"linux/amd64"} Kustomize Version: v4.5.4 console $ cat /etc/os-release $ uname -a Linux bmwiedemann-factory 5.18.6-1-default SMP PREEMPTDYNAMIC Thu Jun 23 05:46:18 UTC 2022 () x8664 x8664 x86_64 GNU/Linux </details<detailsgo1.18 </details<details</details<details</details
/sig release /area build-release
can you please post a link to the full logs from the build?
any news on this? Haven't heard from (context: we worked to fix this a couple of times in )
Unfortunately not, is this something we can fix on the OBS side? If the go build timestamp () is the main issue here then we can approach a fix in our build pipeline.
It would be good if would not be embedded in the first place. It might be possible to replace it somehow in our builds, but all other people wanting reproducible builds, would need to replicate that.
full logs from the build: from kubernetes-1.24.3 built 2022-08-18
can you please try setting GOCACHE=off when you run the build? I am suspecting that the temp directory is being generated here: (or add an export in for that env var)
I to but still got these diffs.
So i built golang from source, peppered it with changes to track down where this is getting created and found that it is coming from: I don't know yet how to fix it. will dig more when i can (travelling for the next few days)
Would it work to override the , I see that they use the : Ref: https://reproducible- Edit: Probably not, we may make it configurable in golang :thinking:
this is so tricky :( will dig more when i get back from my event this week
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close not-planned [1]:
Closing this issue, marking it as "Not Planned".
<detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This is still relevant even with 1.27. It might be worth revisiting this issue again. /reopen /triage accepted
Reopened this issue. <detailsIn response to : This is still relevant even with 1.27. It might be worth revisiting this issue again. /reopen /triage accepted Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle rotten
latest run -
I found that our package has the same issue, so it probably it is a toolchain issue in golang - and much faster to debug outside of the slow kubernetes build. =filed
AIUI, the conclusion by the golang people seems to be that commit is not the right approach and one should use with the go call instead. Not sure how that interacts with other parts of your build.
cc (please see referenced commit above!)
I see. I inherited the gcflags before so I plead innocence. It seems plausible to change it from to - I don't know when I will have time to look at that, though.
My patch doesn't seem to fix the whole problem: thockin-glaptop4 kubernetes buildtrimpath /$ make kubelet +++ [1114 19:10:12] Building go targets for linux/amd64 (non-static) thockin-glaptop4 kubernetes buildtrimpath /$ md5sum output/bin/kubelet output/bin/kubelet thockin-glaptop4 kubernetes buildtrimpath /$ make kubelet +++ [1114 19:12:40] Building go targets for linux/amd64 (non-static) thockin-glaptop4 kubernetes buildtrimpath /$ md5sum output/bin/kubelet output/bin/kubelet thockin-glaptop4 kubernetes build_trimpath /$
diff --- a 2023-11-14 19:13:45. -0800 +++ b 2023-11-14 19:13:56. -0800 -1,4 +1,4 -38rZHdQckYF9EEzmndR/BAXOwzCcq0Mk1UHT62ha/QLiRgXP4oojWh3oWP9/TWG7KW90IL8vKt1UOEPR +LeLArvl7sgWAd55ROKN2/BAXOwzCcq0Mk1UHT62ha/QLiRgXP4oojWh3oWP9/cYb5ZGQwa6js077YiAvK /lib64/ld-linux-x86- AUATUSH []AA] -,8 +,8 go1.21.3 runtime/cgo: call frame too large -2023-11-15T03:12:40Z -2023-11-15T03:12:40Z +2023-11-15T03:13:47Z +2023-11-15T03:13:47Z pthreadcreate failed: %s bad stack bounds: lo=%p hi=%p expand 32-byte kexpand 32-byte k
That seems unrelated. I tested with and it made our openSUSE package fully reproducible. Your diff contains a timestamp - maybe would get rid of that. The top part of the diff is just the go build-id hash of the remainder and can be ignored.
Thanks for confirming
Will there be backports to 1.24++ ? I found it applies cleanly to 1.26++ and works with some small tweaks in 1.24 and 1.25. If not, I'd submit the patch for our openSUSE packages.
1.24 and 1.25 reached EOL and we don't support these branches any longer. We can cherry-pick this to 1.26+ if folks ( agree
Cherry-picks: 1.29 1.28 1.27 1.26