jiebi/CodeConvo huggingface dataset:
Code change:
"description": "EndpointPort represents a Port used by an EndpointSlice", "properties": { "name": { <del> "description": "The name of this port. All ports in an EndpointSlice must have a unique name. If the EndpointSlice is dervied from a Kubernetes service, this corresponds to the Service.ports[].name. Name must either be an empty string or pass IANA_SVC_NAME validation: * must be no more than 15 characters long * may contain only [-a-z0-9] * must contain at least one letter [a-z] * it must not start or end with a hyphen, nor contain adjacent hyphens Default is empty string.", </del> <ins> "description": "The name of this port. All ports in an EndpointSlice must have a unique name. If the EndpointSlice is dervied from a Kubernetes service, this corresponds to the Service.ports[].name. Name must either be an empty string or pass DNS_LABEL validation: * must be no more than 63 characters long. * must consist of lower case alphanumeric characters or '-'. * must start and end with an alphanumeric character. Default is empty string.", </ins> "type": "string" }, "port": {

Related discussion:
text: What happened: No endpoints created in the EndpointSlice for my service due to the port on the service being too long. Logs show: EndpointSlice created has no endpoint: What you expected to happen: Service that works with Endpoints should work with EndpointSlice. If this is intentional, it should be clearly documented How to reproduce it (as minimally and precisely as possible): Create a service with a long port name Environment: Kubernetes version (use ): 1.16 Cloud provider or hardware configuration: kind
/sig networking
The label(s) cannot be applied. These labels are supported: <detailsIn response to : /sig networking Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network These SIGs are my best guesses for this issue. Please comment if I am incorrect about one. I am a bot run by vllry. ‍
For a rough measure of impact, Istio had 5 ports that were over this limit (and many more that were right at 15 characters), so I suspect it may not be terribly rare.
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry. ‍
This needs to be resolved before endpointslices graduate from alpha
/assign
/remove-triage unresolved
Code change:
TEMP_DIR:=$(shell mktemp -d) ifeq ($(ARCH),amd64) <del> BASEIMAGE?=k8s.gcr.io/build-image/debian-base:buster-v1.3.0 </del> <ins> BASEIMAGE?=k8s.gcr.io/build-image/debian-base:buster-v1.4.0 </ins> endif ifeq ($(ARCH),arm) <del> BASEIMAGE?=k8s.gcr.io/build-image/debian-base-arm:buster-v1.3.0 </del> <ins> BASEIMAGE?=k8s.gcr.io/build-image/debian-base-arm:buster-v1.4.0 </ins> endif ifeq ($(ARCH),arm64) <del> BASEIMAGE?=k8s.gcr.io/build-image/debian-base-arm64:buster-v1.3.0 </del> <ins> BASEIMAGE?=k8s.gcr.io/build-image/debian-base-arm64:buster-v1.4.0 </ins> endif ifeq ($(ARCH),ppc64le) <del> BASEIMAGE?=k8s.gcr.io/build-image/debian-base-ppc64le:buster-v1.3.0 </del> <ins> BASEIMAGE?=k8s.gcr.io/build-image/debian-base-ppc64le:buster-v1.4.0 </ins> endif ifeq ($(ARCH),s390x) <del> BASEIMAGE?=k8s.gcr.io/build-image/debian-base-s390x:buster-v1.3.0 </del> <ins> BASEIMAGE?=k8s.gcr.io/build-image/debian-base-s390x:buster-v1.4.0 </ins> endif RUNNERIMAGE?=gcr.io/distroless/static:latest

Related discussion:
text: The based container images for are shipped with the wrong architecture within the manifest if not amd64: Refers to We updated the images a few minutes ago to buster-v1.3.0, which are now ready to be used as base for the multi-architecture images. If the images contain the right architectures, then it may magically happen that the kube-proxy images are fixed, too. This is because we already use the build syntax for it: /cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig release
Code change:
if podConfig.NS == "" { return nil, fmt.Errorf("Cannot create pod with empty namespace") } <del> if len(podConfig.Command) == 0 && !NodeOSDistroIs("windows") { </del> <ins> if len(podConfig.Command) == 0 { </ins> podConfig.Command = "trap exit TERM; while true; do sleep 1; done" }

Related discussion:
text: <!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --ci-kubernetes-e2e-gci-gce-serial (gce-cos-master-serial) Four tests: Kubernetes e2e suite.[sig-storage] CSI Volumes [Driver: csi-hostpath] [Testpattern: Dynamic PV (default fs)] subPath should unmount if pod is gracefully deleted while kubelet is down [Disruptive][Slow][LinuxOnly] Kubernetes e2e suite.[sig-storage] In-tree Volumes [Driver: hostPath] [Testpattern: Inline-volume (default fs)] subPath should unmount if pod is gracefully deleted while kubelet is down [Disruptive][Slow][LinuxOnly] Kubernetes e2e suite.[sig-storage] In-tree Volumes [Driver: hostPathSymlink] [Testpattern: Inline-volume (default fs)] subPath should unmount if pod is gracefully deleted while kubelet is down [Disruptive][Slow][LinuxOnly] Kubernetes e2e suite.[sig-storage] CSI Volumes [Driver: csi-hostpath] [Testpattern: Dynamic PV (default fs)] subPath should unmount if pod is force deleted while kubelet is down [Disruptive][Slow][LinuxOnly] is failing since June 29th 2021 (but the first failure was different than more recent ones). Commit , infra commit , . Other three tests are failing since July 9th 2021. Commit , infra commit , . Testgrid: https://k8s- Prow history: For CSI Volumes: For In-tree Volumes: There is another open issues for failing tests in this job: Failures reported there are about missing metrics, and were occuring before unmounting problems. So I opened a new issue, even though it's the same SIG. These tests don't look related to me, but I don't have enough context, maybe they somehow are. /sig storage /kind failing-test /milestone 1.22
You must be a member of the GitHub team to set the milestone. If you believe you should be able to issue the /milestone command, please contact your and have them propose you as an additional delegate for this responsibility.
Code change:
if podConfig.NS == "" { return nil, fmt.Errorf("Cannot create pod with empty namespace") } <del> if len(podConfig.Command) == 0 && !NodeOSDistroIs("windows") { </del> <ins> if len(podConfig.Command) == 0 { </ins> podConfig.Command = "trap exit TERM; while true; do sleep 1; done" }

Related discussion:
text: <detailsIn response to : <!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --ci-kubernetes-e2e-gci-gce-serial (gce-cos-master-serial) Four tests: Testgrid: https://k8s- Prow history: For CSI Volumes: For In-tree Volumes: #### Anything else we need to know: /kind failing-test /milestone 1.22 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/milestone v1.22
/cc
/cc (my close/reopen yesterday was accidental, sorry)
I can take a look /assign
The non-subpath counterpart of this test is passing: These tests seem to use the default storageclass, whereas failing tests use in-tree hostpath and csi-hostpath
The CSI ones are failing because the hostpath driver never received a NodeUnstage (whereas it did in passing tests), causing the driver to error on volume deletion.
In the failed run kubelet has this error: which seems to have prevented kubelet from triggering a UnmountDevice. This error didn't exist in a successful run.
Hostpath driver can run with and without support for attach. Do you know which mode is enabled for this test?
Between the last passing time and when the test started failing, are there any PRs merged that look suspicious?
According to csi-attacher logs, attach is disabled for both the good run and the failing run. Haven't spotted any obvious PRs so far, but I did find this: in the successful run, the log message for the successful is this: and the failed run: so the reconstructed volume names are somehow different. Checking reconstruction logic
This merged 12 days ago and was a pretty big change to pod lifecycle and could have trickling impact to volume manager. Does the timing of test failure match?
There is also this reconstruction fix that merged yesterday and seems related to outer vs inner spec name. I don't know if that will help fix this issue:
cc
Last successful run started at 7/8 17:49 PT, first failing run started at 7/9 2:23 PT.
Code change:
if podConfig.NS == "" { return nil, fmt.Errorf("Cannot create pod with empty namespace") } <del> if len(podConfig.Command) == 0 && !NodeOSDistroIs("windows") { </del> <ins> if len(podConfig.Command) == 0 { </ins> podConfig.Command = "trap exit TERM; while true; do sleep 1; done" }

Related discussion:
text: was merged at 7/8 16:48 PT, so it's possible (if there's a delay in picking up master) Also this is in the failing kubelet logs but not in the successful one: I0720 18:20:42. ] "Reconciler sync states: could not find volume information in desired state, clean up the mount points" podName=-e508-45c2-8509- volumeSpecName="pvc--a615-4e89-afe8-"
From , the newly introduced log message here exists in the failing kubelet logs but not the successful one:
Failures starting 07/08: /sig node for visibility
I think caused the failures. Before after kubelet restarts, pod will still be in desired state (even through it is in terminating state), so that clean up mounts can be done completely. I think now, terminating pods will not be back to desired state, but reconstruction to actual state cannot completely clean up mounts. I am testing it about this.
Is there a reason why reconstruction to actual state cannot clean up mounts? Note that a pod can be force deleted at any time, so a terminating pod that is force deleted would still have to be capable of being cleaned up (it won't be in desired state)? (note too that the expectation of the changes in is that terminating and fully terminated and "never seen before" should all completely equivalent from the perspective of other loops in the kubelet) If I understand you correctly, is this an existing bug that we're now triggering consistently?
We have both "grace pod deletion" and "force pod deletion" test cases that are failing.
There were a couple of key changes to volume loops in : Roughly before was "it looks like it's got no running containers", and after my change is "we are going to be stopping all the containers from now on". So this change meant that once a pod starts terminating, we are in "tearing down mode" and so my understanding of this method was that "any pod where containers are being torn down is not in the desired state of the world".
Code change:
if podConfig.NS == "" { return nil, fmt.Errorf("Cannot create pod with empty namespace") } <del> if len(podConfig.Command) == 0 && !NodeOSDistroIs("windows") { </del> <ins> if len(podConfig.Command) == 0 { </ins> podConfig.Command = "trap exit TERM; while true; do sleep 1; done" }

Related discussion:
text: After a restart, we would definitely not add a pod back that was in terminating state into desired state of the world, but "restart + this change" should be roughly equivalent to "restart and force delete while restart is processing" The other change was in which changed a live CRI read (which was trying to enforce that the dsowp is synchronized with the pod worker) into a threadsafe check against the pod worker directly (the pod worker can provide authoritative answers to "this pod has no running containers" now). The latter change seems less likely to impact you.
Thank you for the details! I think your change is making the right change, but unfortunately some of volume management still have issues, and so the tests rely on some behaviors that is not very right (pod volume is back to desired state even after it is in terminating) The cleanup mount currently does not have the logic to clean up global mount point if the volume plugin sets it up. I am still testing more and will share more details.
is this a release blocker?
Yes, this impacts volume cleanup if kubelet restarts for any reason.
On a kube-up cluster running the latest master against the latest e2e binary, I was able to reproduce the in-tree failures but not the CSI ones for some reason. Will focus on in-tree failures for the time being
/triage accepted
marking release-blocker per note on slack:
/priority critical-urgent (once upon a time we used to not ship a release if any priority/critical-urgent issues were open, not sure if we decided to stop doing that)
We were able to find the trigger for the CSI failures - on 6/29 there was a change in the hostpath driver test manifest, and the driver's is enabled. This makes the driver return an error instead of only logging a warning message on if the volume isn't unstaged properly. IIUC there's a long standing issue that reconstruction doesn't handle node unstage. could you confirm? If that's the case, then these CSI failures are not release blocking. Furthermore, the CSI test with graceful delete started to fail on 7/9 because of since graceful and force deletes are now considered the same from the point of view of the volume manager. cc
When that check was enabled, I don't think we realized it would impact the reconstruction path; it was enabled to validate a different issue.
Code change:
if podConfig.NS == "" { return nil, fmt.Errorf("Cannot create pod with empty namespace") } <del> if len(podConfig.Command) == 0 && !NodeOSDistroIs("windows") { </del> <ins> if len(podConfig.Command) == 0 { </ins> podConfig.Command = "trap exit TERM; while true; do sleep 1; done" }

Related discussion:
text: Are we running into the issue that reconstruction of staging directories isn't supported () OR is it that our reconstruction path requires drivers to have real mount points ()? Either way since both are known gaps with reconstruction, I think it makes sense to to disable the flag until we can address those issues.
I was looking at this today too, trying to understand the underlying cause. That is why the failures are being reported now, but the part that I didn't understand is why NodeUnstageVolume is not being called for these volumes. In this instance: The DeleteVolume request fails because a staged path is still present: And in the kubelet log, NodeUnpublishVolume is called Then we hit this error: And NodeUnstageVolume is never called for this volume (same for other failures I looked at) so DeleteVolume keeps failing. So I think you're right that it's triggered by reconstruction, and it looks similar to , except in this case the restart happened before NodeUnpublishVolume.
And it's probably not since the test uses a PVC
Will look deeper into why UnmountDevice wasn't issued as well, but some update on the in-tree tests: the force delete test is skipped for the in-tree hostpath CSI driver because of Graceful tests are now failing as well because they behave like force tests due to
I don't think the issue is related to PVC or not. I think it's related to drivers that do not actually mount devices as a separate filesystem than the root.
There are three possible flows during kubelet restart pod gracefully delete case, the pod still exists (it is not considered as terminated in function isPodTerminated(); note that this logic has some changes after , see , so pod volume will be back to desired state, and reconciler will try to set up volume and fill in actual state. When pod volume is finally deleted from desired state, reconciler will trigger teardown with information from actual state to unpublish and unstage volume. pod is forcefully deleted, pod volume will not be in desired state. If volume reconstruction succeeds (be able to reconstruct volume spec), volume will be back into actual state. Then reconciler will trigger teardown with information from actual state to unpublish and unstage volume (this part is similar to case 1).
Code change:
if podConfig.NS == "" { return nil, fmt.Errorf("Cannot create pod with empty namespace") } <del> if len(podConfig.Command) == 0 && !NodeOSDistroIs("windows") { </del> <ins> if len(podConfig.Command) == 0 { </ins> podConfig.Command = "trap exit TERM; while true; do sleep 1; done" }

Related discussion:
text: last case is reconstruction failed (in this csi-host-path test, reconstruction will fail due to issue related to , it fail to check mount point for csi hostpath driver because it is using a bind mount of the same device ). In this case, reconciler will try to clean up mounts directly, but only published volume path, not the staged volume path (this logic is not implemented due to missing information). As a workaround, we can now disable check-volume-lifecycle if it does not cause other issues. Then we will work on to make sure csi-host-path driver can pass volume mountpoint check during reconstruction.
This issue is now mitigated with merged. There are two longer-term issues that are isolated to hostpath plugins (and maybe other plugins that are similar in behavior, perhaps emptydir and similar?): related to IsLikelyNotMountPoint as mentioned. Because these don't impact the core volume machinery and are not regressions, this issue doesn't need to be release blocking. We'll close this issue once (2) is tracked.
is the IsLikelyNotMountPoint issue
after is merged, two csi-hostpath driver subpath test can pass, and two in-tree driver gracefully delete tests were skipped. https://k8s-
can you please cherry-pick to release-1.22 branch?
Opened cherry-pick, please review:
- what is the plan for reenabling them?
cherry-pick merged!
Cheng and I will work on solving IsLikelyNotMountPoint issue for hostpath drivers. That should allow test to pass and we can enable them.
Can this be removed from the milestone now that the workarounds have been merged, while leaving this issue open to track further work?
We can use existing issue to track the real fix. /close
Closing this issue. <detailsIn response to : We can use existing issue to track the real fix. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Code change:
continue } <del> // Remember the last supported test for subsequent test of beta API betaTest = &test </del> ginkgo.By("Testing " + test.Name) suffix := fmt.Sprintf("%d", i) test.Client = c

Related discussion:
text: <!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --ci-kubernetes-e2e-gci-gce-slow [sig-storage] Dynamic Provisioning DynamicProvisioner [Slow] should provision storage with different parameters 5/13 https://k8s- v1beta1.StorageClass got removed in We should remove the v1beta1 test.
/assign can you help take a look?
/triage accepted
/sig storage
I run the test with the latest from master without removing the beta test and it failed with a different error before getting to the beta test part: Do you have any idea what would've happened in my setup? I redeployed k/k using master HEAD
For this testcase: I saw that the provisioned PD has a size of 1.5Gi according to kubernetes: And in gcloud It's really interesting that the test was expecting 2Gi to be created, is this something that was changed recently?
The logs you posted are showing pd csi as the provisioner. Were you testing with the intree driver or csi driver? The CSI driver should to the nearest Gb. I believe intree does the .
I think I did intree because I didn't send as an argument in my command: It's interesting that the PVC has this annotation too:
mmm something interesting about my setup is that I created the cluster with CSIMigration turned on Do you think that might be forcing the usage of ?
Yes likely. Try with CSIMigration off and see if that helps. If so, then this potentially seems like a rounding bug with the csi driver
Thanks I tried with and it did indeed round up to 2Gi.
Code change:
local local_ssds="" if [[ ! -z ${NODE_LOCAL_SSDS+x} ]]; then <ins> # The NODE_LOCAL_SSDS check below fixes issue #49171 # Some versions of seq will count down from 1 if "seq 0" is specified if [[ ${NODE_LOCAL_SSDS} -ge 1 ]]; then </ins> for i in $(seq ${NODE_LOCAL_SSDS}); do <del> local_ssds="$local_ssds--local-ssd=interface=SCSI " </del> <ins> local_ssds="$local_ssds--local-ssd=interface=SCSI " </ins> done <ins> fi </ins> fi local network=$(make-gcloud-network-argument 

Related discussion:
text: Is this a BUG REPORT or FEATURE REQUEST?: BUG /kind bug What happened: If you are using a mac and GCE and you specify NODELOCALSSDS=0 when using GCE via kube-up or e2e --up, then you end up getting 2 local SSDs. What you expected to happen: I would expect to get 0 local SSDs by default and when I specify NODELOCALSSDS=0 explicitly. How to reproduce it (as minimally and precisely as possible): On mac execute: NODELOCALSSDS=0 go run -- -v --up Anything else we need to know?: I've discovered that this issue is due to If NODELOCALSSDS=0, this evaluates to $(seq 0) From man seq on mac osx This was run on mac with the seq manpage indicating it comes from BSD Feb 19 2010. Note: some unix/linux variants, e.g. a test on ubuntu with coreutils 8.25 and a test on centos-7 with coreutils 8.22, had a different behavior that does not demonstrate this bug. Environment: Cloud provider or hardware configuration**: GCE OS (e.g. from /etc/os-release): mac osx
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig storage
Code change:
command: - /bin/sh - -c <del> - echo -998 > /proc/$$$/oom_score_adj && kube-proxy {{api_servers_with_port}} {{kubeconfig}} {{cluster_cidr}} --resource-container="" {{params}} 1>>/var/log/kube-proxy.log 2>&1 </del> <ins> - kube-proxy {{api_servers_with_port}} {{kubeconfig}} {{cluster_cidr}} --resource-container="" --oom-score-adj=-998 {{params}} 1>>/var/log/kube-proxy.log 2>&1 </ins> {{container_env}} {{kube_cache_mutation_detector_env_name}} {{kube_cache_mutation_detector_env_value}}

Related discussion:
text: In cluster/saltbase/salt/kube-proxy/kube-proxy.manifest, we have this: But kube-proxy also has a flag. Is there a reason to use the approach instead of using the flag? cc as you wrote the code, in
/cc who's working on moving kube-proxy to a DaemonSet
In general, oomscoreadj is not expected to be directly updated by pods and kubelet is expected to manage it. If kube-proxy runs in the Guaranteed QoS class, then it's OOM score adj will be set to a low number. Until that happens, kube-proxy process may want to be shielded from OOM kills to an extent (this matters only when kube-proxy's memory usage memory request) Meanwhile, it does not make sense to have two means to configure the same thing. I'll let decide which option he prefers and sunset the other option.
Thanks for clarifying. At this point, I prefer to use the built-in flag instead of relying on composing bash commands. Sent for this.