jiebi/CodeConvo huggingface dataset:
Code change:
apiService := &v1.APIService{ Spec: v1.APIServiceSpec{ <ins> Group:   "apiservicegroup", Version: "v1", </ins> Service: &v1.ServiceReference{Name: "dummy"}, }, }

Related discussion:
text: Can I get your exact repro steps? suggests that the conflicting paths should be removed before the merge. We always call this function with
One other question is if this is reproducible by just applying a set of yaml/installations or does it require a cluster upgrade to trigger? Thanks!
hit same problem and we cannot install any other new helm plugins. maybe we need to move forward
Here are verbose repro instructions. An apiserver restart on 1.28 will induce the issue.
Hello. I'm still having some trouble reproducing although I'm not 100% sure if the configuration is correct. I'm trying to repro via kind. afaik installing via helm has prometheus adapter serving . seems to suggest that prometheus adapter is serving under . Applying the update to the configmap per the instructions didn't change the apiservice on my end. Not sure if this is my misconfiguration or intended. I've also tested via integration test that having two openapis serving the same path doesn't cause errors and one will get filtered out. A simple workaround is probably to delete the metrics-server if it is already superseded by the prometheus-adapter.
The is not set up in an resource for us, we only created an APIService with . When I go fetch the openAPI schema served up by the endpoint, it has schema information for in it if that is enabled using . If I remove then the schema fetched from the server doesn't have stuff in it, and the error is resolved. We have been keeping around because I think something uses it directly, or used to. Maybe ? That could have changed But in any case, this problem is quite severe when it happens so it would be nice to fix it as there may be others who do the same update and have to do some deep diving into the logs and all kinds of crazy diagnostics and guesswork to figure it out.
How does the get reached without an APIService? The resources would be unavailable for the apiserver to query and it'll be through something else? I see, adding resources for me doesn't populate the OpenAPI. I wonder if this could be cloud provider specific. I'm totally supportive of fixing, it's just quite difficult to reproduce on my part .
Hmm good question.
Code change:
apiService := &v1.APIService{ Spec: v1.APIServiceSpec{ <ins> Group:   "apiservicegroup", Version: "v1", </ins> Service: &v1.ServiceReference{Name: "dummy"}, }, }

Related discussion:
text: In our staging cluster I tried bringing back the in the custom metrics server configuration and restarting all the masters, and the problem did not come back, everything seems OK now. So it maybe isn't that easy to reproduce
After updating the configmap, restarting the prometheus-adapter pod to pickup the changes and restarting the apiserver are both required to induce the issue. I have been able to deterministically reproduce this following those steps, please follow up here if you are still having issues.
That's good to hear. I'll need some more help with reproducing it. I've been able to verify that restarting the prometheus adapter pod picks up the configmap changes but restarting the apiserver hasn't caused any crashes for me. I've tested this both on the latest master branch and 1.28.
While I'm yet to fully understand these changes, the 2 PRs below (both of which went into 1.28) seem likely candidates for a regression here: - introduced lazy loading in openAPI v2 handler to load/compute the aggregator spec when requested by client - switched the spec merging function from to any clue if one of the above changes could be causing mischief?
Also from example request at the top, the impact seems to be to the v2 API: Nathan - when you get a chance, could you also confirm if v1 API doesn't run into this?
OpenAPI only supports v2 and v3.
hi, any clue about this issue?
It's possible but I haven't been able to reproduce it. Can I ask if this is happening on cloud providers outside AWS?
Here are the apiservices and relevant pod objects with a working repro from these steps If it's useful, I can provide yamls for these objects in addition to the settings applied in the prometheus-adapter configmap. Can you confirm that the pods backing your prometheus-adapter and metrics-server are running?
I meet this problem
We the problem in GCP as well.
ACK in Alibaba Cloud also meets this problem
We were able to fix the v2 availability by and re-installing the kube-prometheus helm chart. Unfortunately, after restarting the k8s api server, v2 availability is broken again.
yes, I'm running on kind. I'll do some more digging, assuming it's related to those PRs linked by but still haven't pinpointed anything.
Code change:
apiService := &v1.APIService{ Spec: v1.APIServiceSpec{ <ins> Group:   "apiservicegroup", Version: "v1", </ins> Service: &v1.ServiceReference{Name: "dummy"}, }, }

Related discussion:
text: I tried to reproduce on my local setup with based on steps + , but I could still get a successful response when I executed . Then I reverted one of the step mentioned in to and waited until the new pod came alive and then did a apiserver restart and I was able to reproduce the same thing on as well i.e; I get the 503 response when I execute following command after the fact. After this point, even if i change it back to and restart the apiserver , the error is persistent. Let me know if this works for you to reproduce ? PS: Also, i want to call out that there is no issue with , its .
Thanks, alright I found a way to reproduce it consistently without restarting the apiserver. Still investigating the root cause. Thanks for the detailed reproduction and
Quick update: This issue was a combination of bugs both in k/k and prometheus adapter. I opened to mitigate on k/k side which should fix the issue. The issue was difficult to reproduce because it requires a very specific ordering of OpenAPI merging to trigger, which is why I suspect many people were able to reproduce after some restarts to kube-apiserver. Prometheus and metrics server serve differing OpenAPIs for the endpoint. Prometheus also doesn't seem to be registering discovery's OpenAPI properly (missing OpenAPI for but having the OpenAPI for whereas it is all present in metrics server. This inconsistency in how prometheus and metrics server handle the OpenAPI surfaced the merging issue.
a little wired question: why do we need to restart kube-apiserver to reproduce this problem?
Tried to reproduce the issue after cherry picking the fix and didn't run into below error. Fix is working as expected.
It can look like restarting prometheus-adapter "fixes" the issue? I am also curious about suggestion of removing the section from the config map works as a temporary workaround until released upstream (we use and cannot delete the apiregistration)? As I understand the fix for this issue is currently in
Code change:
fi source "${KUBE_HOME}/kube-env" <ins> if [[ -n "${KUBE_USER:-}" ]]; then if ! [[ "${KUBE_USER}" =~ ^[-._@a-zA-Z0-9]+$ ]]; then echo "Bad KUBE_USER format." exit 1 fi fi </ins> config-ip-firewall create-dirs ensure-local-ssds

Related discussion:
text: under kubetest.Up Nov 22 2023, last 7 runs failed No response /sig
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
APIServer failed for
Yes it is failing for me too!
/sig testing
the generator config source must include the env to handle the "external" provider deprecation: /sig cloud-provider /area provider/gcp
/remove-sig testing /sig release
all other jobs seem green in the 1.29 blocking/informing, post that change. i will try sending a PR for this, but you need to find a non-USA approver if you want it merged before Monday. /assign
never mind, the generator is giving me errors: /unassign
/cc
Code change:
// from the registered device plugins. func (m *ManagerImpl) Allocate(node *schedulercache.NodeInfo, attrs *lifecycle.PodAdmitAttributes) error { pod := attrs.Pod <ins> devicesToReuse := make(map[string]sets.String) </ins> // TODO: Reuse devices between init containers and regular containers. for _, container := range pod.Spec.InitContainers { <del> if err := m.allocateContainerResources(pod, &container); err != nil { </del> <ins> if err := m.allocateContainerResources(pod, &container, devicesToReuse); err != nil { </ins> return err } <ins> m.podDevices.addContainerAllocatedResources(string(pod.UID), container.Name, devicesToReuse) </ins> } for _, container := range pod.Spec.Containers { <del> if err := m.allocateContainerResources(pod, &container); err != nil { </del> <ins> if err := m.allocateContainerResources(pod, &container, devicesToReuse); err != nil { </ins> return err } <ins> m.podDevices.removeContainerAllocatedResources(string(pod.UID), container.Name, devicesToReuse) </ins> } m.mutex.Lock()

Related discussion:
text: I noticed that you have already add the limitation into the , so I think it's already the decision that we add the limitation as a short-time solution in v1.9 So I posted a PR to add the check in code,, PTAL :)
Use Case: Let say there is a which node has two Devices of a resource type X and the device IDs are X1 and X2. User pod has one init container, lets call it cont-init and one regular container, lets call it cont-reg. Both cont-init and cont-reg requests 2 unit of resource X. cont-init will run some vendor specific code on allocated Devices and then cont-reg will use these 'initiated' devices. I think this is the use case that mentioned. And i feel 2) can handle this. 2) approach will allocate both the devices, X1 and X2, to both the containers. When the containers start, init container, cont-init, will be run first and will run to completion and here devices will get initialized. Strictly after this cont-reg will be started and now this regular container has initialized devices. Where i am wrong?
I guess in RenaudWasTaken 's case, the initialization task is considered to be managed in the plugin, not in init-container, and in init-container we only use the device, so then we might need to call plugin to do recycle action before the device reused by reguler containers. In practical use, both of the two scenarios may exist, it depends on the request of user. Besides, if we don't issue an Allocate gRPC call, we could not ensure a ”complete reuse“: say an init-container requests 2 devices, and we have two regular containers, each requests 1 device, then due to the lack of runtime info re-generation, the devices could only be reused by one of the containers, not both of them.
It is the init containers' job to initialize things and if in some special case device plugin is doing the initialization part, in that case, init containers will not be needed and serve no purpose in pod spec. Specific to example of memory scrubbing, i think this also makes more sense to be done in init container. Once Device's memory is scrubbed, it can be used by regular container. Let try to run your use case with approach 2).
Code change:
// from the registered device plugins. func (m *ManagerImpl) Allocate(node *schedulercache.NodeInfo, attrs *lifecycle.PodAdmitAttributes) error { pod := attrs.Pod <ins> devicesToReuse := make(map[string]sets.String) </ins> // TODO: Reuse devices between init containers and regular containers. for _, container := range pod.Spec.InitContainers { <del> if err := m.allocateContainerResources(pod, &container); err != nil { </del> <ins> if err := m.allocateContainerResources(pod, &container, devicesToReuse); err != nil { </ins> return err } <ins> m.podDevices.addContainerAllocatedResources(string(pod.UID), container.Name, devicesToReuse) </ins> } for _, container := range pod.Spec.Containers { <del> if err := m.allocateContainerResources(pod, &container); err != nil { </del> <ins> if err := m.allocateContainerResources(pod, &container, devicesToReuse); err != nil { </ins> return err } <ins> m.podDevices.removeContainerAllocatedResources(string(pod.UID), container.Name, devicesToReuse) </ins> } m.mutex.Lock()

Related discussion:
text: Answering on behalf of and Solution 1 makes sense for the moment Solution 2 defeats the purpose of the device plugin, which is setting up a device before making it available to a container. solution 4 seems to be a sane solution. As mentioned in my previous comment, as long as devices are exclusive, I expect the devices exposed in my container to always be in the same state at the start of my container. When / If we implement a mechanism for sharing a GPU / devices across containers then the scenario described by is something that we can have. It doesn't however fit the current situation.
Solution number 1 seems totally reasonable short-term. Long term, simplifying and documenting the relationship between plugin callbacks and container lifecycle would help plugin authors.
For reference, I implemented option 2 solution in PR The change itself is small. This is an alternative short-term fix we may consider to get into a 1.9 patch release. I think the main benefit of taking this fix compared to Option 1 solution is that we can change the underlying implementation to support init containers later with less user-facing impact.
No mater we take PR or PR for 1.9, I think we need to have this issue approved for 1.9 milestone. or or can any of you approve this issue for 1.9? Thanks!
Code change:
importsBelowBase := strings.HasPrefix(imp, i.BaseDir) importsAllowed := false for _, allowed := range i.AllowedImports { <del> importsAllowed = importsAllowed || strings.HasPrefix(imp, allowed) </del> <ins> exactlyImportsAllowed := imp == allowed importsBelowAllowed := strings.HasPrefix(imp, fmt.Sprintf("%s/", allowed)) importsAllowed = importsAllowed || (importsBelowAllowed || exactlyImportsAllowed) </ins> } return importsBelowRoot && !importsBelowBase && !importsAllowed

Related discussion:
text: uses everywhere, directly and via client-go. On the other hand, we have in :
any idea why the import checker does not catch this?
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
Code change:
if w.skipIfBlocked { select { case w.destChan <- t: <del> w.fired = true </del> default: } } else { w.destChan <- t <del> w.fired = true </del> } if w.stepInterval > 0 {

Related discussion:
text: <!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I was writing a variant of clock.FakeClock, and noticed a couple of bugs in . Look at the pointer comparison in --- the two pointers will never be equal, but the author clearly expected equality when is the index of the fakeClockWaiter corresponding to the fakeTimer. The two pointers will never be equal because one is a pointer into the fakeClock's slice while the other pointer is to a field of the . The consequence of this bug is that the Stop function is ineffective. Also, determines the returned value based on the original copy of the waiter, which --- being a copy --- did not get updated if/when the waiter fired. Note also that does not exercise FakeClock::NewTimer at all. What you expected to happen: I expected FakeClock Timers to work properly. How to reproduce it (as minimally and precisely as possible): Use the TestFakeTimer func introduced in Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
The fake timer's Reset() method also does not work correctly.
Code change:
EmptyDir: &v1.EmptyDirVolumeSource{SizeLimit: &sizeLimit}, }, v1.ResourceRequirements{}), }, <ins> { evictionPriority: 0, // This pod should not be evicted because it uses less than its limit pod:              diskConsumingPod("emptydir-disk-below-sizelimit", useUnderLimit, nil, v1.ResourceRequirements{Limits: containerLimit}), }, </ins> }) }) })

Related discussion:
text: Is this a BUG REPORT or FEATURE REQUEST?: Bug Report /kind bug What happened: improved disk eviction testing to make it more precise. For example, it expects a pod which uses 299 Mb of disk not to be evicted if it sets its limit to 300 Mi. As shown on , this test passes for all OS distributions except CoreOS. I have ruled out an issue with the storage driver and backing filesystem since both COS and CoreOS use overlay backed by extfs. What you expected to happen: I expected the pod using 299Mb not to be evicted when its limit is 300 Mb. I expected not to see this event: How to reproduce it (as minimally and precisely as possible): The container runs this command to use 299Mb: Each iteration writes 1Mb to disk, producing a total of 299 Mb of disk usage. The pod has set to 300Mi. The test has the LocalStorageCapacityIsolation feature gate enabled. Anything else we need to know?: This may be a discrepancy in the (which we use to measure container disk usage) or , which is used by the test to write exact quantities of disk.
/sig node
cc can you take a look at this after thanksgiving?
I bumped the coreos image... going to see if it clears up the flake
It appears the root filesystem is being included in the disk space calculation . Do you think the coreos image might be slightly larger than the other images under test?
One more code snippet that is interesting:
thats the container's root filesystem
correct. the feature appears to add and compare to the quota: The reproducible steps do not take into account the rootfs space. If the quota is set to 300Mi, then the under loop count would have to take into account the rootfs space to not exceed the limit. I do not see anywhere in the tests or the reproduce steps this part of the calculation.
The refers only to the writable layer of the container, and excludes base (image) layers. With the provided command, the only files that are in the container's rootfs are the ones written by the command, and should amount to 299Mb. Additionally, the container does not write any logs (through stdout), so that value should be 0.
Code change:
EmptyDir: &v1.EmptyDirVolumeSource{SizeLimit: &sizeLimit}, }, v1.ResourceRequirements{}), }, <ins> { evictionPriority: 0, // This pod should not be evicted because it uses less than its limit pod:              diskConsumingPod("emptydir-disk-below-sizelimit", useUnderLimit, nil, v1.ResourceRequirements{Limits: containerLimit}), }, </ins> }) }) })

Related discussion:
text: I ran a quick test on a vm with the current image, and found that both and correctly wrote and measured disk space correctly. I also tested as it inside a container, and verified that we are measuring the container's writable layer correctly. My guess now is that it has something to do with how we calculate disk usage by logging with journald. I am looking into it
Actually, the previous comment was incorrect. When I ran the test, I found that each file was 1028 kb instead of 1024 kb, which resulted in the command writing 300.3 Mb instead of 299 Mb. I will change the test to require slightly less precision. This appears to be because an empty file in coreos takes up 4 kb. Test this by running . On other OS we test on, empty files use 0 kb. This is what accounts for the difference.
I tracked down an issue to the dd call. The stdout and stderr need a redirect to /dev/null to suppress the following messages: Problem Line: Fix:
I PRed that fix and changed it to stderr, since the other tests seem to use stderr.
leaving this open until we get a green run
I'll continue to test locally to see if I find anything else.
opened PR with the help of and CoreOS defaults to using the actual size instead of the apprent size.
Code change:
} } <del> func stringptr(val string) *string { return &val } </del> // overlyCautiousIllegalFileCharacters matches characters that *might* not be supported.  Windows is really restrictive, so this is really restrictive var overlyCautiousIllegalFileCharacters = regexp.MustCompile(`[^(w/.)]`)

Related discussion:
text: The discovery cache for kubectl is defaulted to 10 minutes: That means that every 10 minutes of using against a cluster, it runs API requests against all group versions, which can take quite some time on clusters with many operators installed (like , , , etc.). I previously wrote that shows that every 10 minutes a simple makes 170 API requests against a test cluster, vs 4 API requests when the cache exists. That results in significant productivity implications against such clusters. Here is a time comparison between a request that runs the discovery cache vs one that doesn't: The discovery cache doesn't really have to run every 10 minutes, as CRDs don't change that often. A lot of unnecessary load is created on clients and servers. This issue is raised to change the discovery cache default to 24 hours and also to make the discovery cache ttl configurable. Running the discovery cache every 10 minutes has a significant productivity impact on using on clusters with many CRDs as it is taking time to run these unnecessary requests. The discovery cache doesn't really have to run every 10 minutes, as CRDs don't change that often. A lot of unnecessary load is created on clients and servers.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cli
I would like to work on this issue.
/assign