jiebi/CodeConvo huggingface dataset:
Code change:
rm -rf "${LOCAL_OUTPUT_BINPATH}" mkdir -p "${LOCAL_OUTPUT_BINPATH}" <del> # Remove the container if it is left over from some previous aborted run "${DOCKER[@]}" rm -f -v "${KUBE_BUILD_CONTAINER_NAME}" >/dev/null 2>&1 || true </del> <ins> kube::build::destroy_container "${KUBE_BUILD_CONTAINER_NAME}" </ins> "${docker_cmd[@]}" bash -c "cp -r ${REMOTE_OUTPUT_BINPATH} /tmp/bin;touch /tmp/finished;rm /tmp/bin/test_for_remote;/bin/sleep 600" > /dev/null 2>&1 # Wait until binaries have finished coppying

Related discussion:
text: Github automatically adds a artifact to every release, which sounds very similar to the artifact that we generate as the release artifact. Assuming we don't have control over the naming or presence of the autogenerated source code artifact, I think we should consider renaming our published artifact to better disambiguate between the two.
We distribute most of our artifacts outside of GitHub. We should probably stop uploading this one entirely.
cc
We've got a thread going on w/ Tim. /assign
can you please link so folks can follow the bread crumbs? ( cc )
-- It's currently sensitive. Will share once there's an outcome.
ACK. thanks for the heads up
I think it's safe to discuss openly at this point. We had a report that the sha512 hashes on our release artifacts were incorrect. We have confirmed that the mismatch was due to a bug in the release pipeline, and not malicious. This issue here was just a point of confusion that came out of that discussion.
thanks !
/assign
Going one step back since I think the exists for historical reasons: The client and server binaries were part of this bundle, which is not the case any more: So it seems that is mostly a collection of licenses and scripts. Do we really need it or may we cleanup some code paths and remove it completely?
+1 to put the word out and remove it completely if no one depends on it.
I'm not confident that CI is not using these versioned scripts, in fact i'm fairly sure that it is. However it would not be pulling it from github.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale I think this remains as relevant as ever.
/milestone v1.23
/triage accepted
Hi and k8s 1.23 bug triage shadow here.
Code change:
rm -rf "${LOCAL_OUTPUT_BINPATH}" mkdir -p "${LOCAL_OUTPUT_BINPATH}" <del> # Remove the container if it is left over from some previous aborted run "${DOCKER[@]}" rm -f -v "${KUBE_BUILD_CONTAINER_NAME}" >/dev/null 2>&1 || true </del> <ins> kube::build::destroy_container "${KUBE_BUILD_CONTAINER_NAME}" </ins> "${docker_cmd[@]}" bash -c "cp -r ${REMOTE_OUTPUT_BINPATH} /tmp/bin;touch /tmp/finished;rm /tmp/bin/test_for_remote;/bin/sleep 600" > /dev/null 2>&1 # Wait until binaries have finished coppying

Related discussion:
text: There hasn't been any activity or linked PRs for this issue. Just wanted to check in to see if this is still on track for 1.23.
/milestone v1.24
Hi I am part of the bug triage team for v1.24. We noticed that this issue hasn't been updated for a while. Just wanted to check if there is any work planned that would go into this.
this one keeps moving down the milestones. i don't see any activity since november. So kick the can down the road /milestone v1.25
Hi My name Hossein Salahi from Bug-Triage team. I am just checking to ensure if everything is on track for K8s 1.25. Thanks!
That's a question for or
any remarks on that?
the release 1.25 bug triage Lead. I'm reaching out to see the status of this issue and if we are still targeting the current release. Also, kindly if you can update the priority so we can track it in the future.
this one keeps moving down the milestones yet again!. i don't see any activity since i moved this down the line. So kick the can down the road: /milestone clear (instead of v1.26 as this has been moving down the line for a few releases now)
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
What's the current status of this issue? Is there anything that we can do to move this forward?
We got new conversations going on during Kubecon on whether we should publish our release binaries on GitHub. Given that and I think we should rethink now what we publish on the release pages. I think we should start publishing all binaries for one. It is a common practice everywhere and I think users largely expect them to be there.
Code change:
rm -rf "${LOCAL_OUTPUT_BINPATH}" mkdir -p "${LOCAL_OUTPUT_BINPATH}" <del> # Remove the container if it is left over from some previous aborted run "${DOCKER[@]}" rm -f -v "${KUBE_BUILD_CONTAINER_NAME}" >/dev/null 2>&1 || true </del> <ins> kube::build::destroy_container "${KUBE_BUILD_CONTAINER_NAME}" </ins> "${docker_cmd[@]}" bash -c "cp -r ${REMOTE_OUTPUT_BINPATH} /tmp/bin;touch /tmp/finished;rm /tmp/bin/test_for_remote;/bin/sleep 600" > /dev/null 2>&1 # Wait until binaries have finished coppying

Related discussion:
text: let's scope the issue to the actual "Kubernetes Source Code" tarball. We can still remove it from what I can see from the discussion.
I agree with Removing the "Kubernetes Source Code" tarball should be as easy as removing this from krel: However, there are a few questions: is it safe to do so? I believe yes, because this "Kubernetes Source Code" tarball is pointing to the GCS bucket, and we'll still continue publishing that tarball to the bucket The link is also available in the changelog, so it's easy to find do we need any deprecation period? Considering the above two points, I don't think so. If you folks agree with that, I can go ahead and create a PR to update krel.
I agree on that, let's drop an additional note on k-dev that others are aware of the change.
/assign
Update: An announcement to proceed with the removal of the "Kubernetes Source Code" artifact has been sent to k-dev: PR to remove this artifact from being published: Removal will go into effect starting the January patch releases.
Code change:
w.initialValue = results.Unknown } <del> podName := getPodLabelName(w.pod) </del> basicMetricLabels := metrics.Labels{ "probe_type": w.probeType.String(), "container":  w.container.Name, <del> "pod":        podName, </del> <ins> "pod":        w.pod.Name, </ins> "namespace":  w.pod.Namespace, "pod_uid":    string(w.pod.UID), }

Related discussion:
text: The metric used to report acutal pod in the label , but now I see the podname prefix only. The full pod name in the label, so I can trace the metric to the failing pod. Check raw api and verify the actual pod name with uids. <details</detailsAzure AKS <details</detailsNo response No response No response_
/sig node
/triage accepted /priority important-soon /cc As discussed this is a change in functionaloty that wasn't highlighted in release notes. Need to decide on what do do with it. I understand it's a change of functionality. what is the side effect of this for you? Does the label without last part not informational enough?
I've been using the metric in my custom grafana dashboard that shows CPU, RAM, IO usage of pods with graph of increases in probes. If the apps are stable, the probe graph looks steadily. I can see if some irregularities in system metrics are influencing probes of respective pod. Also, if readiness probe is failing, the pod is removed from service, but is not restarted so I must have knowledge of pods with failing readiness probes. I attach the screenshot of my graph. ! So I'd welcome the possibility to pair the probes with exact pod and thank you for looking into it.
are we thinking that this resulted in an undesirable change? Spologies for being a bit late to reply, I'm returning from a brief paternity leave staycation.
This issue is labeled with but has not been updated in over 90 days, and should be re-triaged. Important-soon issues must be staffed and worked on either currently, or very soon, ideally in time for the next release. You can: Confirm that this issue is still relevant with (org members only) Deprioritize it with or Close this issue with For more details on the triage process, see /remove-triage accepted
Well this is awkward... I still believe the actual behavior does not respect the intended idea...
This issue makes it hard to link probe metrics with right pods. The value of pod label in and are also inconsistent with how it is used in other kubelet metrics, see: vs Definitely a bug, regression since 1.25.
I can't get anyone to look at the fix/revert: you may have better luck
/milestone v1.28
discussion in
You've pushed it through, congrats!
Code change:
sched.Recorder.Eventf(victim, preemptor, v1.EventTypeNormal, "Preempted", "Preempting", "Preempted by %v/%v on node %v", preemptor.Namespace, preemptor.Name, nodeName) } <del> metrics.PreemptionVictims.Set(float64(len(victims))) </del> <ins> metrics.PreemptionVictims.Observe(float64(len(victims))) </ins> } // Clearing nominated pods should happen outside of "if node != nil". Node could // be nil when a pod with nominated node name is eligible to preempt again,

Related discussion:
text: <!-- Please only use this template for submitting enhancement requests --What would you like to be : Change scheduler metric from Gauge to Histogram Why is this needed: is recorded for each preemption attempt. A histogram is better. /sig scheduling /help /good-first-issue
/help /good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I can work on this issue.
/assign
I could work on this as well if doesn't have time to work on this.
I am working on it, I will create the initial PR in less than a week. Sorry for the delay.
Code change:
var ( // Master nodes are not added to standard load balancer by default. defaultExcludeMasterFromStandardLB = true <ins> // Outbound SNAT is enabled by default. defaultDisableOutboundSNAT = false </ins> ) // Config holds the configuration parsed from the --cloud-config flag

Related discussion:
text: <!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: is the recommended way to configure the outbound networks together with Azure standard loadbalancer (SLB). However, for load balancer rules are disabled by default. That means all frontend IPs would be used as the SNAT IPs for virtual machines, according to . This would introduce troubles for various security scenarios, as there's no way to get a stable list of egress IPs. Azure cloud provider should support this scenario, and disable the outbound SNAT when outbound rules are configured by provisioning tools. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Others: /kind bug /sig azure /priority critical-urgent
Code change:
a.cachedCreds = newCreds // Only close all connections when TLS cert rotates. Token rotation doesn't // need the extra noise. <del> if a.onRotate != nil && oldCreds != nil && !reflect.DeepEqual(oldCreds.cert, a.cachedCreds.cert) { a.onRotate() </del> <ins> if len(a.onRotateList) > 0 && oldCreds != nil && !reflect.DeepEqual(oldCreds.cert, a.cachedCreds.cert) { for _, onRotate := range a.onRotateList { onRotate() } </ins> } return nil }

Related discussion:
text: <!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Data race in client-go with below sample code: Data race: What you expected to happen: No data race How to reproduce it (as minimally and precisely as possible): Using above given sample code Kubernetes version (use ):Cloud provider or hardware configuration:OS (e.g: ): OSX
/sig api-machinery
I'd like to take it. /assign
/remove-sig api-machinery /sig auth cli
Bug triage for release 1.17 here. Since this issue has been tagged for the milestone, we would like to check it's status. Code freeze is on Nov. 14. Will this issue be resolved before then?
Another reminder here that code freeze is in less than two weeks (Nov. 14). We will need to move this issue out of the milestone if isn't going to be resolved before then.
Code change:
releaseHugepages() ginkgo.By("restarting kubelet to pick up pre-allocated hugepages") <del> restartKubelet() </del> <ins> // stop the kubelet and wait until the server will restart it automatically stopKubelet() </ins> waitForHugepages() })

Related discussion:
text: ci-kubernetes-node-kubelet-serial-hugepages when start the pod with the resources requests that contain multiple hugepages resources should set correct hugetlb mount and limit under the container cgroup Unexpected error: "pod ran to completion", Multiple sizes huge pages is going to be graduated to GA, so having this test job fixed is quite important.
/sig node
/assign
/triage accepted
/cc
I did a small analysis and I can see that the pod failed to start, I can not find any failure messages under the kubelet log, so it really hard to say why it failed.
were you able to reproduce it locally? btw, there is a "right way" to fix this - to fix dynamic update of a hugepage resources. This would eliminate need in a kubelet restart. There is even a PR for that: , we just need to make it working again.
after conversation with on slack I'd say we should stick to the current procedure - restarting kubelet when hugepages allocation changes. So, we should investigate why kubelet doesn't restart properly in this test case and fix it.
Nope, I failed to reproduce it locally, also under the CI, it does not appear often. The best thing will be to describe the pod in case if it failed, so we will see the reason.
under the code we have so from the log , I can see that the kubelet received the pod before the kubelet stopped, so the problem that pod dropped to Failed state because of the kubelet stop.
So some additional details, the one who responsible to restart the kubelet again, is the server service: restart the kubelet service are waiting for hugepages to be available are starting the pod server service restart the kubelet again because it was not healthy for too long - failed to start because of the kubelet restart
great findings! Should we just stop the kubelet and let the server service to start it again? We can also check if the kubelet is restarted successfully before starting the pod.
I tried it under the PR , also I unsure how can we check if the kubelet restarted successfully. All tests that I wrote before I checked if the node has NotReady state and waited until it will be Ready, but in case of the restart probably the node will always have Ready state because the time period is very short. I will continue to investigate and will check locally how can I fix it.
Code change:
releaseHugepages() ginkgo.By("restarting kubelet to pick up pre-allocated hugepages") <del> restartKubelet() </del> <ins> // stop the kubelet and wait until the server will restart it automatically stopKubelet() </ins> waitForHugepages() })

Related discussion:
text: So the only reliable solution that I found, it just trying to start the pod number of times. I tested it 5 times and the test passed. Can you please review
Can we access kubelet service from hugepage test cases? If we can we can use its APIs like or to check if kubelet is ready after the restart. Overall I have a feeling of duplication test/e2enode/services functionality in This is a main reason for this and similar bugs I believe. It would be nice to get rid of this duplication if possible. That would nicely isolate kubelet service controls in one place and eliminate conflicts like this one. wdyt?
not really, the used mostly to start all needed components to emulate the cluster(not only the kubelet) and the test does not have access to it. I think you can separate the readiness check under the importable method, but it worth the separate PR.
I updated the PR to check the healthiness of the kubelet and looks like it works just fine.
thank you. it looks good to me so far. I'll add lgtm label when is merged and tests are all green.
the job is still flaky. can you reopen this issue?
I opened a different one - , the issue can affect any job, not only the hugepages one.
I can open a fix to W/A an issue(start the pod until it will succeed), it should fix all problems, WDYT?
i think it's ok as a workaround, until is fixed
would it be better to wait until api server reports correct resource value before running the pod?
We are waiting for it, the problem that the scheduler and the kubelet see a different picture regarding resources requested by pods.
ok. In this case restarting the pod until it succeeds seems reasonable to me to fix the flakiness until is fixed.
any progress on the workaround fix?
Code change:
containers: - name: fluentd-cloud-logging image: gcr.io/google_containers/fluentd-gcp:1.6 <ins> resources: limits: cpu: 100m </ins> env: - name: "FLUENTD_ARGS" value: "-qq"

Related discussion:
text: During debugging , I noticed that one of services pods: monitoring and logging pods stuck at pending sometimes forever: I logged into the node, and found the container is running happily on the node: Looks like status is never reported from kubelet. I checked kubelet log, then I found: Why we cannot delete mirror pod? and Created a new one? Because this regression, once we run into such state, the rest of test won't be triggered, and eventually timeout / failed.
checks those service pods in ready state, and prevent the test to run once it is pending. cc/ I think this is the real issue I talked to you earlier about PodStatus.
I spoke to offline. For some reason, kubelet thinks the spec of the mirror pod does not match the static pod locally. It then decides to delete the mirror pod and recreate it, repeatedly. As a result of that, it slows down (due to the QPS limit?) reporting statuses of other pods. This can be reproduced by running the e2e "Shell" test. I am looking into why the specs would suddenly mismatch. Did any recent PR modified the pod spec in apiserver?
The recently introduced default cpu share limit was not there in the static pod, causing the difference between its spec and the mirror pod's.
Seeing this as well. Makes the whole cluster slow and unusable for me. Sent from my iPhone
Is there a desired fix for this?
said she would add default resource limits to the static pod manifest files. If we go for that, I think it's reasonable to fail (not run the static pod) if resource limit is not specified. Also, the resource limits set in the mirror pods are not meaningful because kubelet creates containers based on the spec of the static pod. In general, static pods do not subject to any defaulting values applied at the apiserver.
Pretty sure that this is fixed by . Certainly testing against head since then works well for me. What version are you testing?
was fixing node status, and the problem had been around for a while. This issue is about pod status, which is still unfixed on head. These two are very different, and my cluster did include the fix :) This issue will slow down kubelet at times, but if you get lucky and the pod status (running) get sent out in time, the tests will still pass.
Code change:
containers: - name: fluentd-cloud-logging image: gcr.io/google_containers/fluentd-gcp:1.6 <ins> resources: limits: cpu: 100m </ins> env: - name: "FLUENTD_ARGS" value: "-qq"

Related discussion:
text: There is no way the issue is fixed by that PR. I am still seeing the issue on HEAD. The system gets worse the longer its up. I agree with the analysis of the issue in source PR Sent from my iPhone
I am on this one, and here is proposed solution in my mind so far (not final yet since I haven't put too much thoughts into it yet): 1) For all our static pods I am going to update them with some initial cpu limit for now. This should quickly stop flakiness caused by this particular issue. 2) Next step is that for static pods, if there is no initial cpu limit, kubelet should refuse to run. Before I do this, I need to send an announcement to our container-vm users about this potential breakage for the in the future. Another potential fix is we can ignore resource changes when comparing if a mirror pod is updated or not. I think is proposed a similiar one earlier. But I had a concern with it: this is defeat the initial purpose we created mirror pods and sync back to master. So if one made some resource changes on the static pods, you have to always ssh-ed into the node to see if the new value applied or not. Also eventually (post 1.0) those mirrored pod's resource limit is going to be used by scheduler when calculating if user pods are fit or not, we need them to be accurated. Another concern is in the code implementation itself to distinuguish mirror pods future from rest of pods. I guess this shouldn't be the big issue here. Any more suggestions here? One puzzle I had is that why master component pods doesn't have issue at all. I manually restart kube-apiserver pod, still doesn't have the issue.
Depending on time we could just remove the default limit from the limit range as a stop gap. Sent from my iPhone
Can you just bypass defaulting at the apiserver when you create mirror pods? I can't think of any other solution that works in general and is easily maintainable. Approach 2 you described in the previous entry works if we are defaulting cpu limit, but breaks as soon as we default anything else.