jiebi/CodeConvo huggingface dataset:
Code change:
By("verifying service-disabled is still not up") framework.ExpectNoError(framework.VerifyServeHostnameServiceDown(cs, host, svcDisabledIP, servicePort)) }) <ins> It("should be rejected when no endpoints exist", func() { namespace := f.Namespace.Name serviceName := "no-pods" jig := framework.NewServiceTestJig(cs, serviceName) nodes := jig.GetNodes(framework.MaxNodesForEndpointsTests) labels := map[string]string{ "nopods": "nopods", } port := 80 ports := []v1.ServicePort{{ Port:       int32(port), TargetPort: intstr.FromInt(80), }} By("creating a service with no endpoints") _, err := jig.CreateServiceWithServicePort(labels, namespace, ports) if err != nil { framework.Failf("Failed to create service: %v", err) } nodeName := nodes.Items[0].Name podName := "execpod-noendpoints" By(fmt.Sprintf("creating %v on node %v", podName, nodeName)) execPodName := framework.CreateExecPodOrFail(f.ClientSet, namespace, podName, func(pod *v1.Pod) { pod.Spec.NodeName = nodeName }) execPod, err := f.ClientSet.CoreV1().Pods(namespace).Get(execPodName, metav1.GetOptions{}) framework.ExpectNoError(err) serviceAddress := net.JoinHostPort(serviceName, strconv.Itoa(port)) framework.Logf("waiting up to %v wget %v", framework.KubeProxyEndpointLagTimeout, serviceAddress) cmd := fmt.Sprintf(`wget -T 3 -qO- %v`, serviceAddress) By(fmt.Sprintf("hitting service %v from pod %v on node %v", serviceAddress, podName, nodeName)) expectedErr := "connection refused" if pollErr := wait.PollImmediate(framework.Poll, framework.KubeProxyEndpointLagTimeout, func() (bool, error) { _, err := framework.RunHostCmd(execPod.Namespace, execPod.Name, cmd) if err != nil { if strings.Contains(strings.ToLower(err.Error()), expectedErr) { framework.Logf("error contained '%s', as expected: %s", expectedErr, err.Error()) return true, nil } else { framework.Logf("error didn't contain '%s', keep trying: %s", expectedErr, err.Error()) return false, nil } } else { return true, errors.New("expected wget call to fail") } }); pollErr != nil { Expect(pollErr).NotTo(HaveOccurred()) } }) </ins> }) // TODO: Get rid of [DisabledForLargeClusters] tag when issue #56138 is fixed.

Related discussion:
text: I have a cluster from HEAD as of 2 weeks ago: So indeed it seems to not work. If I hit that same service IP from the VM itself I immediately get curl: (7) Failed to connect to 10.0.199.54 port 80: Connection refused. If I insert an iptables jump to KUBE-SERVICES at the front of FORWARD, it correctly rejects. This seems in accordance with the iptables flowchart. I'll do some spelunking to see if there's a reason NOT to reject in this case. Vallery Lancey <:
Can you check IPVS mode?
This is why we need tests. Sigh.
Proposed fix is - still needs your e2e :)
Thanks Tim, if I have time I’ll get a PR up tonight.
Sure will check IPVS mode.
Base on my test, IPVS has no such issue.
Code change:
} klog.Errorf("Failed to GetNodeNameByIPConfigurationID(%s): %v", ipConfigurationID, err) <del> errors = append(errors, err) </del> <ins> allErrs = append(allErrs, err) </ins> continue } nodeResourceGroup, nodeVMSS, nodeInstanceID, nodeVMSSVM, err := ss.ensureBackendPoolDeletedFromNode(nodeName, backendPoolID) if err != nil { <del> klog.Errorf("EnsureBackendPoolDeleted(%s): backendPoolID(%s) - failed with error %v", getServiceName(service), backendPoolID, err) errors = append(errors, err) </del> <ins> if !errors.Is(err, ErrorNotVmssInstance) { // Do nothing for the VMAS nodes. klog.Errorf("EnsureBackendPoolDeleted(%s): backendPoolID(%s) - failed with error %v", getServiceName(service), backendPoolID, err) allErrs = append(allErrs, err) } </ins> continue }

Related discussion:
text: <!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --/kind bug /sig cloud-provider /area provider/azure Azure in-tree CCM fails to delete a load balancer: The corresponding logs: The clouprovider config is: The Nodes are actually Azure VMs that belong to a VMO (aka vmss with flexible orchestration mode) - regular VMs and managed independently of the vmss. So those VMs are managed on Azure as . LoadBalancer deletion works with the very same setup with v1.21.1 of the in-tree Azure CCM. After recent cherry-picks, with v1.21.4 we observe the failure above. a cluster using VMs that belong to a VMO a Service of type LoadBalancer the Service of type LoadBalancer sure that the deletion fails with: Kubernetes version (use ): v1.21.4 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Gardener Network plugin and version (if this is a network-related bug): Others:
/assign /triage accepted
Code change:
optf(dm) } <del> // initialize versionCache with a updater dm.versionCache = cache.NewVersionCache(func() (kubecontainer.Version, kubecontainer.Version, error) { return dm.getVersionInfo() }) // update version cache periodically. if dm.machineInfo != nil { dm.versionCache.UpdateCachePeriodly(dm.machineInfo.MachineID) } </del> return dm }

Related discussion:
text: The recently merged PR a version cache in dockertools, which makes the test  flaky. The reason is that injected error into fakedockerclient and then expected to see the error from right after that (See ). However, the error was consumed by the goroutine which periodically updates the version cache (See ). The docker daemon version is rarely updated, and we only need version once for each container creating, during which we also do container create, start, inspect #etc. 1) Do we care about the cost of the call so much that we have to introduce the complexity? 2) Kubelet periodically calls , can we rely on that to update the cache? Anyhow, at least, we should disable the cache or have a fake one in the unit test. :) /cc
I once considered move this cache to runtime level, while thus may need to change runtime interface. What if only cache apiversion?
We can only get via , so there will still be the data race. :) One option is to update the version cache in , now kubelet periodically call to check the runtime status. Another better option is to cache the version in layer, now is under our control, we can refactor it with an additional argument indicating whether to use cached value or get directly from the daemon. If we cache the version only for performance, I prefer the second option which is cleaner and more efficient. :)
OK, will look into this tonight
The unit test itself should not start a goroutine to update the cache (unless you're actually testing the goroutine). We should disable it anyway, as we often do in our unit tests. It also sounds like we don't need a goroutine dedicated for this. If we can tolerate the additional overhead from time to time, we can set a expiration time (e.g., 1 minute), and only call the runtime if the entry expires.
Totally agree. :) Maybe a new function in wrapping the update logic, or wrapping the logic in the .
expiration time sounds good in this case
This is happening quite frequently - increasing priority to P0.
Code change:
# Get the compute project project=$(gcloud info --format='value(config.project)') <ins> project=${project// /} </ins> if [[ ${project} == "" ]]; then echo "Could not find gcloud project when running: `gcloud info --format='value(config.project)'`" exit 1

Related discussion:
text: The compute project and zone in the prerequisites check are incorrectly calculated: !!! Error in hack/make-rules/test-e2e- Error in hack/make-rules/test-e2e- 'tee -i "${artifacts}/build-"' exited with status 1 0 Call stack: 1: hack/make-rules/test-e2e- main(...) Exiting with status 1 blocked by the following errors in each case: 1- Could not find gcloud compute/zone when running: .... 2- Could not find gcloud project when running: ..... 1- gcloud config unset compute/zone 2- gcloud config unset project 3- make test-e2e-node REMOTE=true LIST_IMAGES=true gcloud --version Google Cloud SDK 453.0.0 alpha 2023.10.27 beta 2023.10.27 bq 2.0.98 core 2023.10.27 gcloud-crc32c 1.0.0 <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node /assign
Code change:
defer func() { framework.DeletePodWithWait(f, f.ClientSet, pod) }() <del> err = framework.WaitForPodRunningInNamespace(f.ClientSet, pod) Expect(err).To(HaveOccurred(), "while waiting for pod to be running") </del> By("Checking for subpath error event") selector := fields.Set{

Related discussion:
text: <!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: gce-new-master-upgrade-cluster-new-parallel Which test(s) are failing: Varies, but all subpath failures, including: [sig-storage] CSI Volumes [Driver: csi-hostpath-v0] [Testpattern: Dynamic PV (default fs)] subPath should fail if subpath with backstepping is outside the volume [Slow] [sig-storage] CSI Volumes [Driver: csi-hostpath] [Testpattern: Dynamic PV (default fs)] subPath should fail if subpath directory is outside the volume [Slow] [sig-storage] CSI Volumes [Driver: csi-hostpath-v0] [Testpattern: Dynamic PV (default fs)] subPath should fail if subpath file is outside the volume [Slow] [sig-storage] CSI Volumes [Driver: csi-hostpath] [Testpattern: Dynamic PV (default fs)] subPath should fail if subpath file is outside the volume [Slow] ... and pretty much every other subpath test, but never all of them at once. There's also a few other storage tests failing, such as: [sig-storage] Volume expand [Slow] Verify if editing PVC allows resize [sig-storage] Detaching volumes should not work when mount is in progress Since when has it been failing: 11/22 Testgrid link: https://k8s- Reason for failure: These flakes started around the time that merged, but doesn't match up with the exact merge stamp, so it's probably coincidental. The subpath test failures seem to be mostly timeouts: ... so possibly this is just GCE fail. Anything else we need to know: This test job has always been flaky, with around a 40% failure rate. <!-- DO NOT EDIT BELOW THIS LINE --/kind flake /sig storage /priority important-soon cc
can you please help investigate this. It looks to me like the tests are flaking for and . I wonder if the v0 tests are racing with the non-v0 tests for something in this test?
The other tests including for the other CSI driver and non-CSI in-tree plugins are solid green which also makes me think this is a issue with the hostpath tests or driver.
Code change:
defer func() { framework.DeletePodWithWait(f, f.ClientSet, pod) }() <del> err = framework.WaitForPodRunningInNamespace(f.ClientSet, pod) Expect(err).To(HaveOccurred(), "while waiting for pod to be running") </del> By("Checking for subpath error event") selector := fields.Set{

Related discussion:
text: (just adding additional similar flakes here for tracking) Similar flakes in :
adding to the milestone for triage since the flakes look new
These all 4 subpath tests call and it fails in finding a specific failed event by using . As uses to check if the specific error event happens, and , I guess that tests may flake if other failed event happens before the expected one. Also, already wait for the event, we might be able to delete  in . I will create a PR to fix above.
/priority critical-urgent /remove-priority important-soon /milestone v1.13
while we should update the test with your PR, I'm not sure if it's going to fix the failure. Looking at one failing test log, these are the events I see on the failing pod: So it seems to be failing before even reaching the subpath logic.
Timeline of events: Test pod started getting processed by kubelet: It took a minute after the test pod was created for the csi driver to finish registering and mount to succeed, and then subpath event to trigger: So should address the issue. I was initially confused because the test logs didn't show the subpath event for some reason
I opened up to investigate if 1 minute for csi plugin registration is expected
I've a tracker issue for the complete list of storage flakes: This is so that we can track whether we're addressing most of the flakes with the subpath fixes, or if additional fixes are required.
keeping open until flakes are verified cleaned up
/reopen Reopening this issue until we can verify the fix in CI
As of this morning we are seeing CSI "subpath should fail" flakes in the following job Multiple "subpath should fail" flakes in this slow job alone Multiple "subpath should fail" flakes in the 1.13 slow job, similar to master. The following 2 tests are flaky only in this upgrade job can you help us determine if these are cases are flaky tests that you are fixing or is there an underlying CSI bug that needs addressing?
root cause of this is a delay in an external sidecar containers and the tests not waiting long enough to accommodate the delay. There are no changes required in (outside of test) to handle this. These should not be 1.13 blocking. We'll continue to look in to getting them non-flaky today.
Code change:
defer func() { framework.DeletePodWithWait(f, f.ClientSet, pod) }() <del> err = framework.WaitForPodRunningInNamespace(f.ClientSet, pod) Expect(err).To(HaveOccurred(), "while waiting for pod to be running") </del> By("Checking for subpath error event") selector := fields.Set{

Related discussion:
text: In the meantime, we can mark these tests as flaky to reduce the noise, if you'd like. looks like the same issue () is still being hit. See : And logs from (): We need to make the tests handle this condition more gracefully. The suggestion here is one possible solution.
Would be good to capture this as a known issue in release notes.
to followup on capturing this in Release notes
per , can you create a targeted fix increasing the timeout? Looks like marking these tests as flaky will be difficult for 1.13, but if that doesn't fix it we will need to do that.
Short term fix to bump the test timeout:
As per discussion in release burndown meeting, this is not a 1.13 blocker. please move this issue to 1.14 if there are no pending issues you are concerned about
Ok looks like the test still flaked. Looking at , the mount succeeded after a minute and proceeded to subpath processing, which (correctly) failed. However, for some resaon, the test is not seeing the subpath event and still failed after 4 minutes. On kubelet, subpath processing failed after a minute and generated the event: But the test doesn't see it for some reason, and still times out after 4 more minutes:
I saw the same on CSI volume mount succeeded at: Kubelet correctly failed the pod because of subpath But test still failed: Because the test failed to find the pod event. This looks like a test issue. Not a 1.13 blocker.
can you PTAL at this
Is there a way to check if events are overloaded and not able to handle new events? The MountVolume retries while the driver is not registered is very frequent and could have overloaded the event server.
Checking for the existence of an event seems like the wrong way to test this. Would checking pod status alone be insufficient? Ultimately that is what we really care about.
Checking for the specific event guarantees that we are failing with the correct error. But perhaps events are not reliable enough to depend on. Checking if the Pod never came up running would probably reduce the flakes but could potentially mask false negatives. cc for any thoughts.
Code change:
defer func() { framework.DeletePodWithWait(f, f.ClientSet, pod) }() <del> err = framework.WaitForPodRunningInNamespace(f.ClientSet, pod) Expect(err).To(HaveOccurred(), "while waiting for pod to be running") </del> By("Checking for subpath error event") selector := fields.Set{

Related discussion:
text: Tests are still flaking and actually it seems like in general storage flakes in slow are flaking more since 11-21 or so: https://k8s- My theory is that the driver registration delay is causing a flood of events, which causes tests that check for events to fail. I think this is revealing multiple issues in both the tests and product, and we should work on fixing them all, although any one of the fixes should mitigate the test flakes: Driver registrar can sometimes take a minute to come up: Volume manager operation executor doesn't seem to be throttling retries or backing off: Tests should not rely on events: Tests should wait for CSI driver to be registered on all nodes before creating test pods:
and have been merged and will address the biggest issues. has been backported to 1.13 already, backport is still pending.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Code change:
case "${KUBE_OS_DISTRIBUTION}" in ubuntu|coreos) <del> echo "Starting cluster using os distro: ${KUBE_OS_DISTRIBUTION}" >&2 </del> source "${KUBE_ROOT}/cluster/aws/${KUBE_OS_DISTRIBUTION}/util.sh" ;; *)

Related discussion:
text: Example: kubectl cluster-info shows a redundant line: The cluster is already up, so the phrase "Starting cluster" is confusing in context of the command. (My kubernetes test cluster is running on AWS i.e. KUBERNETES_PROVIDER=aws. I had used kube- to create a test kubernetes cluster on aws). Log of my shell session:
Thanks for the bug report I've filed a PR to fix it.
Code change:
func (r *LogREST) NewGetOptions() (runtime.Object, bool, string) { return &api.PodLogOptions{}, false, "" } <ins> // OverrideMetricsVerb override the GET verb to CONNECT for pod log resource func (r *LogREST) OverrideMetricsVerb(oldVerb string) (newVerb string) { newVerb = oldVerb if oldVerb == "GET" { newVerb = "CONNECT" } return } </ins>

Related discussion:
text: Is this a BUG REPORT or FEATURE REQUEST?: What happened: use kubectl search log with follow option get api latency (apiserverrequestlatenciessummary{quantile="0.99",resource="pods",verb="GET"} )high to give a alert in my cluster. the apiserver log is below and we can see it costs 1m25s. What you expected to happen: do not make this api latency account in getting pod request api latency How to reproduce it (as minimally and precisely as possible): use kubectl to search a pod log with follow option long enough and you will get this metric apiserverrequestlatenciessummary{quantile="0.99",resource="pods",verb="GET"} higher than the ordinary case. Anything else we need to know?: I am happy to fix it. Environment: Kubernetes version (use ): 1.6 Cloud provider or hardware configuration**: bare-metal OS (e.g. from /etc/os-release): ubuntu 16.04 Kernel (e.g. ): 4.4 Install tools: kubeadm Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig API Machinery
/area apiserver
/sig api-machinery
/assign Thanks... looking forward to your PR!
fixed with PTAL
Code change:
) const ( <del> // When these values are updated, also update cmd/kubelet/app/options/options.go // A copy of these values exist in e2e/framework/util.go. </del> <ins> // When these values are updated, also update cmd/kubelet/app/options/container_runtime.go // A copy of these values exist in test/utils/image/manifest.go </ins> currentPodInfraContainerImageName    = "gcr.io/google_containers/pause" <del> currentPodInfraContainerImageVersion = "3.0" </del> <ins> currentPodInfraContainerImageVersion = "3.1" </ins> ) // GetServerArchitecture fetches the architecture of the cluster's apiserver.

Related discussion:
text: <!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: the pause-amd64:3.0 image on has an empty schema2 manifest, which prevents is from being pulled by proxy/caches which only support V2 registries. What you expected to happen: Being able to pull ` using a V2 manifest compatible client. Anything else we need to know?: The proxy/cahce is requesting this
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
After talking with , he mentionned the following
AFAIK, does not support well. After inspecting the image, you can find is still using manifest v1.
/cc FYI
Im not sure which exact version x schéma X is needed. What I know is that we can download through the proxy/cache all other Kubernetes kube-system images, only the pause image is giving us trouble. Therefore, I would say that having the image available using the same format as the other kube-system images would be the request here.
Yeah, we haven't updated the pause image in a while, since it doesn't really ever change. I'm not sure whether rebuilding + pushing over the existing tag is safe, or if we should bump to or something. thoughts?
If we do rebuilding/updating the image, that would help and a bunch of similar issues.
it's better/save to go with new tag
I didn't realize there were changes to the pause binary that were never actually deployed. It seems like we should probably just do that (and bump the tag).
that sounds great. Any idea when the existing 3.0 tag could be pushed again?
We shouldn't push over the 3.0 tag; we should push a 3.1 or 4.0 tag instead. I'm not sure who owns the pause container who should drive this. Maybe sig-node and/or
It seems like it should be sig-node. I'll look into it.
The new container has been built, can you test with
Its working! thanks! I was able to pull it through our artifactory proxy , Lee Verberne <:
excellent, thanks for checking. /assign