jiebi/CodeConvo huggingface dataset:
Is this a BUG REPORT or FEATURE REQUEST?: What happened: Now the and has been consistent in by PR . There is no need to convert v1.ObjectReference to clientv1.ObjectReference in . What you expected to happen: Remove this function and cleanup the codes which call this function. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?:
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <bre.g., for API Machinery<br(2) specifying the label manually: <bre.g., for sig/scalability<br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
Will submit the cleanup after PR merged.
/sig api-machinery
/sub
<!-- Please only use this template for submitting enhancement requests --The built-in and cluster roles here are deprecated in 1.13 when this issue was resolved: /sig storage /milestone v1.17 /priorit important-longterm
You must be a member of the GitHub team to set the milestone. If you believe you should be able to issue the /milestone command, please contact your and have them propose you as an additional delegate for this responsibility. <detailsIn response to : <!-- Please only use this template for submitting enhancement requests -- /milestone v1.17 /priorit important-longterm Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc /priority important-longterm
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale /assign
Bug triage for 1.17 here with a gentle reminder that code freeze for this release is on November 14. Is this issue still intended for 1.17?
Which jobs are flaking: ci-kubernetes-node-kubelet-serial-hugepages Which test(s) are flaking: all Testgrid link: https://k8s- Reason for failure:
/sig node
/assign /cc
GitHub didn't allow me to assign the following users: cynepco3hahue. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign /cc Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
testgrid alert:
/cc
I started to look into logs and the issue that one of the kubelet restarts fails on: I will try to reproduce the issue locally.
any news? Have you managed to reproduce the issue locally?
I already have a fix that is waiting for merge -
As the RBAC role need to be related to resources. I think we could use a command to get the supported resources. ref to
This would be cool.
I thought we a command that listed resources in the discover doc a long time ago.
is just the versions, but principle is the same. We want to list all the types, their resources, and tie that back to "explain".
What's the best way to determine this information currently? I'm having difficulty figuring out the exact set of permissions (and how they would be written in an RBAC role) for any given kubectl command. The documentation for this right now is split across multiple places and certain things are not explained at all. Specifically, it's difficult to determine which "API verb" is used for which operations. Take as an example. It's unclear if the way to allow this is to use the "proxy" subresource of a pod or service with "get"/"create"/etc. verbs, or to use the "proxy" verb with the resource (pod, service) directly. is a similar case. The almost works for this purpose, but it's still difficult. Operations for a given resource are grouped into categories like write, read, status, and proxy, but these things don't obviously correlate with API verbs. Again, proxy is especially confusing as there is no mention of a proxy "verb". The API operations are only shown using their HTTP verbs. One thing that could help on this document would be a section on the right column under "kubectl command" that shows exactly the rule needed in an RBAC role to allow this command via kubectl. And even then, that assumes an exact 1-to-1 mapping between API endpoints and kubectl commands, which I doubt is always the case. (For example, port fowarding to a pod seems to require both a GET on the pod resource and a CREATE on a port forward subresource.) There also needs to be better documentation about how this information could have been determined in the first place. I've resorted to manually looking through API requests made with but even then it's difficult to figure out. Amusingly, I was the one who wrote which is referenced in the original issue description (as well as the table explaining verbs in ).
tl;dr: It's too difficult right now to go from "I want a user to be able to run " to "here is the RBAC rule to use for that" for any case where there isn't a specific example shown in the docs.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
It seem that relative PR stop updated. If this issue still valuable, i will help fix it
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
All of our external load balancer tests failed for the better part of a day due to the GCE project that our tests run in hitting its quota for number of target pools. While the target pools have been deleted, the fact that we hit our quota indicates that there must be a leak somewhere, and we need to plug it.
For reference, the previous issue was
Could the leakage be related to ? The quota exhaustion occurred about 22 hrs after that.
I'd guess that this is due to GetTCPLoadBalancer returning that the load balancer doesn't exist if the forwarding rule is gone, ignoring the target pool, while the service controller considers its deleting done if the cloud provider says that the load balancer doesn't exist. I that logic to the cloud provider because the other implementations of DeleteTCPLoadBalancer weren't/won't be idempotent in the sense of returning ok if the LB doesn't exist. To fix it, we'll either need to make sure all Delete implementations are idempotent and remove the Get call in the controller, or add a method along the lines of IsTCPLoadBalancerDeleted to ask the cloud provider whether all pieces of it have been completely cleaned up. We can't make GetTCPLoadBalancer check that because it's relied on to tell us whether or not the balancer is fully created.
Copying over my commentary from on why the GKE e2e tests are leaking so many forwarding rules and target pools vs the GCE tests:
Is there a way for the services test to wait for the service controller to clean up the external resources before finishing (without just doing a sleep)?
Not without polling GCE directly for whether the resources still exist, although that may be worth it as a workaround. What would be perfect is that clean-up-all-cloud-provider-resources API method that we talked about a couple months ago.
Can we not perhaps use GCE namespaces or projects as a cheap way of getting this here?
I don't believe so. We can't use projects due to the fact that projects can't be programatically created (they're intended to be long-lived). I'm not aware of anything along the lines of a GCE namespace other than networks, which won't work because target pools and forwarding rules unfortunately don't seem to be scoped to networks.
The target pool leak in the code might be fixed, but the forwarding rule + target pool leak from deleting GKE clusters too quickly is certainly still happening. Reopening for the sake of that half of the issue.
Can we just add a short sleep to jenkins after the last test has run before it deletes anything?
Yes, you could add a sleep to Ginkgo's AfterEachSuite(): But I'd be concerned about what the upper bound on the sleep period is. Can we get the tests and code in sync with a 1 minute latency here? (e.g. if the controller garbage collects every 60 seconds, then the test only needs to sleep for 60 sec?) And add a BEEG comment to the test's sleep config noting that increasing the sleep time will not improve things, otherwise I'm pretty sure that someone noting flakes in this test in future will think "maybe I should just increase the sleep period a little bit."
Actually, thinking about this a little more, it would be better to put the "sleep" logic in the k8s master teardown procedure (i.e. not in the e2e tests or in Jenkins), because otherwise users are going to run into the same resource leakage problems as we are. So master deletion blocks on cleanup of backend cloud resources.
There's no way to block master deletion from the master itself. The test framework (or user) just pulls the rug out from underneath the master. See
Robby's right -- is the real solution to this, but until we have that the best we can do is to have the wait happen elsewhere. The options seem to be: GKE itself Ginkgo's post-testing logic Jenkins' teardown logic I don't like putting it in GKE, because arbitrarily slowing down our API makes for a poor experience for the majority of people that don't still have resources being deleted. I'm not eager to put it in Gingko either, since won't that sleep also take effect whenever anyone runs an e2e test locally against a dev cluster? Putting it in the Jenkins logic makes the most sense to me. I'll send a PR and cc you.
We may also want to change the kube-down logic to bring down the master and then nodes in parallel to both speed it up and more closely resemble GKE's teardown logic.
I've deleted the old forwarding rules and target pools from the project.
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: kubelet terminated unexpectedly during hack/local-up-cluster kubelet log as a result apiserver cannot find the node to schedule pods What you expected to happen: kubelet runs successfully and hack/local-up-cluster creates a single node cluster How to reproduce it (as minimally and precisely as possible): run hack/local-up-cluster from HEAD Anything else we need to know?: additional cgroups check seemed to be introduced in ; hack/local-up-cluster works when I check out the parent commit have CI for hack/local-up-cluster and , so I'm not sure if it's an error on my side. It would be greatly appreciated if someone could explain it to me Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Debian GNU/Linux Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig node cc
Thanks for raising such a detailed issue! cc any thoughts here? Confusingly, I think the kubelet has been assigned a different cgroup for memory and for cpu... however the fact that its using makes me think the user is relying on system defaults. out of curiosity, have you made any mods to how your system assigns cgroups?
It sounds like we might want to handle the error more gracefully. If we can't get the kubelet's cgroup, we should fall back to collecting metrics from all cgroups.
collecting metrics from all the cgroups on the system? Any concerns about how many cgroups there could be? Also, or any interest in implementing this fix (assuming we go with the fallback behavior)? If not, I can try.
thanks for looking into this! Sorry I'm not familiar with cgroups. I'm don't know any modification to my system assigns cgroups in particular
I can put out a PR this week
xref
Thanks!
/kind failing-test <!-- If this is a flake, please add: /kind flake --/sig autoscaling Looks like it just started happening today (@ 2020-09-15 03:54:19 +0000 UTC) !
cc the namespace from the provided object "kube-system" does not match the namespace "horizontal-pod-autoscaling-830". You must pass '--namespace=kube-system' to perform this operation. the namespace from the provided object "custom-metrics" does not match the namespace "horizontal-pod-autoscaling-830". You must pass '--namespace=custom-metrics' to perform this operation. the namespace from the provided object "custom-metrics" does not match the namespace "horizontal-pod-autoscaling-830". You must pass '--namespace=custom-metrics' to perform this operation.
related fix:
No I don't think that's the fix for this (that is just a little cleanup). Actually I think we have to delete namespace parameter in line : Because before , the will have no actually effect. But now with the change of , it will have namespace : at the end of the command. But the namespace in yaml file is which is different from . We should not have parameter when applying from file because this test didn't have before, right? Related PR:
/assign
In the new scheduler performance testing framework, (bool) is provided to benchmark performance of pods created in an "operator": (L11 below) However, the initialized Pods created in L5~L7 seem to be also benchmarked, although it doesn't specify . (please see the reproduce steps below). I think it's b/c once scheduler starts, the associated Prometheus artifacts are also up, and it's a single in-memory datastore. So no matter we start to collect the metric in which step, the prometheus data is already up-to-date. In the above workload config, the final data have benchmarked 1500 pods instead of 1000 pods. I guess it won't work either if running in the pod initialization step or not. Unless we manage a breakpoint at the end of each step, and calculate delta histogram for the step that specifies . Pods created in step without are not benchmarked. Apply the following diff: And then run: You will get an output sth. like: /sig scheduling
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Could you please help confirm if it's an issue?
cc
Unfortunately, does not apply for metrics gathered from . As you mentioned. The default gatherer collects all metrics produced by the scheduler. It's the scheduler itself who knows pod specs and would have to decide which pods will get ignored when updating metrics. The question is if computing metrics for a subset of pods processed by the scheduler is a valid use case.
This is probably what we want. Is something like that possible ?
I think being able to absolutely ignore certain pods for metrics is an overkill.
The use case is to ignore the pods created in steps that don't specify , so as to collect the Pod metrics create at particular step(s) (). Otherwise, the semantics of option is quite misleading. Nope, that's not the intention. We don't need to target certain pods, like using label selector or whatever. Just as I mentioned, measure pods created at particular steps.
I would do something like the following: if a step has collectMetrics:false, we wait for them to finish scheduling, and then we take a snapshot of the histogram. Then for the next step that has collectMetrics:true, we take the diff with the latest snapshot to report the results. This might not be perfect, as we don't control if the metrics have finished syncing when we take the snapshot. But I think that's fine. Would doing this be possible? Now, if we have steps that run in parallel, one with collectMetrics:false and another with collectMetrics:true, then this doesn't work. Note that I don't know the details of how the gatherer works.
That adheres to what I have in my mind - we can only manage this on client side. I think it's fine. As we are already fetching the "snapshot" when pods get scheduled when equals true. True. An alternative is to design the case to be like: the step with to employ Pods that are all unschedulable, and then specify a custom metric "schedulere2eschedulingdurationseconds[result="unschedulable"]" to target a particular histogram instead of all histograms (default option) inside a histogramVec. Note this can only be possible when and some subsequent changes get in place.
We might just call Histogram.Clear() method which will clear what has been collected so far at the end of phase: Assuming individual phases have no pods in common and each phase can clear metrics without affecting other phases. Though, as said, it will work only for serial case.
WDYT
I'm afraid doesn't work. That was my initial thought as well, but no luck. I guess it's b/c, theoretically, we cannot reset a metric on the server side - Promoetheus' model doesn't support this. But let me know if I miss anything.
The performance tests assume all the metrics all gathered and accessed in the same binary. As long as that's the case, works.
We already call in the code:
That was my understanding as well, but no luck :( Here is a diff to call in case when "concreteOp.CollectMetrics" is false: <details<summaryDiff:</summary<p</p</detailsAnd below is the output - it's still 1500 (500 init pods plus 1000 measured pods) instead of 1000 being benchmarked: <details<summaryOutput:</summary<p</p</details
Also, if you run the tests and together, the result proves doesn't work as expected:
But I do find a way to fix this: .
I create a ResourceQuota with two pods limit in my namespace. Then, I create a statefuleSet with 3 replicas. As expected, two pods are created successfully, and the third pod failed to create. But, the ReadyReplicas is not 2 ,but 1. The ReadyReplicas is correct 1Create a ResourceQuota with two pods limit in a new namespace. 2Create a statefulSet with 3 replicas 3Check if the ReadyReplicas of statefulSet matchs the number of stateful running pods. 1According to the logic of the statefulSet controller, new pod created in current sync is not be immediately counted into the ReadyReplicas. 2Failure to create pod will cause to return error, then skip updating status. <details</details<details</details<detailskind v1.23.3 </details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig apps
/assign
When ResourceClaimParameters or ResourceClassParameters fail validation in resource driver during PodSchedulingContext sync loop, the error is visible only in the events of the PodSchedulingContext. The error should be visible on the object that fails validation as well, ResourceClaim or ResourceClass respectively. Creating ResourceClaim with unsupported parameters' values or unsupported No response <details</details<details</details<details</details<details</details<details</details<details</details
/assign
/sig node
/triage accepted
KEP:
The following PriorityClass endpoints was Promoted it 1.14 without Conformance tests, and remain untested: replace-, read-, patch-, list-, delete-collection- SchedulingV1PriorityClass According to this APIsnoop query, there are still some remaining PriorityClass endpoints which are untested. with this query you can filter untested endpoints by their category and eligiblity for conformance. e.g below shows a query to find all conformance eligible untested,stable,core endpoints ! createSchedulingV1PriorityClass deleteSchedulingV1PriorityClass Promoted to Conformance 1.19 with /sig testing /sig architecture /area conformance
Is the ultimiate goal to use conformance tests to explicitly issue all CRUD requests against PriorityClass? And, is equivalent to "update PriorityClass"?
all APIs must have Conformance tests unless they are not eligible for Conformance due to the There is other APIGroups that have both Replace and Update as endpoints. That is the prerogative of the SIG to decide.
can you please point us to similar conformance tests used for core objects such as pods and nodes?
This is a life cycle test for Create, List, Update, Patch, DeleteCollection More can be found in the test/e2e folder. Also look for examples on the Conformance at previously Promoted Conformance test Reach out if we can help with more information on our approach to Conformance testing.
I remembered about this PR. Last week a few lines of code was to cover the last of the "Lease" endpoint group for conformance. The one file contain the Conformance tests for the complete life cycle of Lease endpoints.
/assign I will work on it.
/reopen Will monitor in test grid to move to Conformance after 2 weeks soak.
Reopened this issue. <detailsIn response to : /reopen Will monitor in test grid to move to Conformance after 2 weeks soak. </details
/triage accepted
Is this a BUG REPORT or FEATURE REQUEST?: /kind bug This is a followup from filed separately now that we're not tracking this as part of 1.9 release. The tests have been disabled with [DisabledForLargeClusters] tag for now, and this fact will be mentioned in the scalability validation report for v1.9. This issue tracks fixing the test so the tag can be removed and the test ran again in the large cluster. /assign /kind e2e-test-failure
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Is this e2e test still broken?
Sorry for the confusion here. Currently this e2e test is disabled on extra large cluster due to some scalability issues with the load balancer related components. At the moment the issues haven't been completely resolved yet.
ok, adding priority so that I know it's not a 1.13 blocker. /priority important-longterm
cc - this is why the relevant tests for are not part of the e2e suite.
This issue is still valid --- some action items on top of my head: Figure out a plan to make service controller properly handle operations in parallel. Check on the status for cloud providers (especially GCP, which the test ran against) on handling large number of nodes as LB backends. /remove-triage unresolved
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or .