jiebi/CodeConvo huggingface dataset:
Code change:
l.migrationCheck.validateMigrationVolumeOpCounts() } <del> ginkgo.It("should write files of various sizes, verify size, validate content [Slow][LinuxOnly]", func() { </del> <ins> ginkgo.It("should write files of various sizes, verify size, validate content [Slow]", func() { </ins> init() defer cleanup()

Related discussion:
text: <detailsIn response to : <!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --ci-kubernetes-e2e-gci-gce-serial (gce-cos-master-serial) Four tests: Testgrid: https://k8s- Prow history: For CSI Volumes: For In-tree Volumes: #### Anything else we need to know: /kind failing-test /milestone 1.22 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/milestone v1.22
/cc
/cc (my close/reopen yesterday was accidental, sorry)
I can take a look /assign
The non-subpath counterpart of this test is passing: These tests seem to use the default storageclass, whereas failing tests use in-tree hostpath and csi-hostpath
The CSI ones are failing because the hostpath driver never received a NodeUnstage (whereas it did in passing tests), causing the driver to error on volume deletion.
In the failed run kubelet has this error: which seems to have prevented kubelet from triggering a UnmountDevice. This error didn't exist in a successful run.
Hostpath driver can run with and without support for attach. Do you know which mode is enabled for this test?
Between the last passing time and when the test started failing, are there any PRs merged that look suspicious?
According to csi-attacher logs, attach is disabled for both the good run and the failing run. Haven't spotted any obvious PRs so far, but I did find this: in the successful run, the log message for the successful is this: and the failed run: so the reconstructed volume names are somehow different. Checking reconstruction logic
This merged 12 days ago and was a pretty big change to pod lifecycle and could have trickling impact to volume manager. Does the timing of test failure match?
There is also this reconstruction fix that merged yesterday and seems related to outer vs inner spec name. I don't know if that will help fix this issue:
cc
Last successful run started at 7/8 17:49 PT, first failing run started at 7/9 2:23 PT.
Code change:
l.migrationCheck.validateMigrationVolumeOpCounts() } <del> ginkgo.It("should write files of various sizes, verify size, validate content [Slow][LinuxOnly]", func() { </del> <ins> ginkgo.It("should write files of various sizes, verify size, validate content [Slow]", func() { </ins> init() defer cleanup()

Related discussion:
text: was merged at 7/8 16:48 PT, so it's possible (if there's a delay in picking up master) Also this is in the failing kubelet logs but not in the successful one: I0720 18:20:42. ] "Reconciler sync states: could not find volume information in desired state, clean up the mount points" podName=-e508-45c2-8509- volumeSpecName="pvc--a615-4e89-afe8-"
From , the newly introduced log message here exists in the failing kubelet logs but not the successful one:
Failures starting 07/08: /sig node for visibility
I think caused the failures. Before after kubelet restarts, pod will still be in desired state (even through it is in terminating state), so that clean up mounts can be done completely. I think now, terminating pods will not be back to desired state, but reconstruction to actual state cannot completely clean up mounts. I am testing it about this.
Is there a reason why reconstruction to actual state cannot clean up mounts? Note that a pod can be force deleted at any time, so a terminating pod that is force deleted would still have to be capable of being cleaned up (it won't be in desired state)? (note too that the expectation of the changes in is that terminating and fully terminated and "never seen before" should all completely equivalent from the perspective of other loops in the kubelet) If I understand you correctly, is this an existing bug that we're now triggering consistently?
We have both "grace pod deletion" and "force pod deletion" test cases that are failing.
There were a couple of key changes to volume loops in : Roughly before was "it looks like it's got no running containers", and after my change is "we are going to be stopping all the containers from now on". So this change meant that once a pod starts terminating, we are in "tearing down mode" and so my understanding of this method was that "any pod where containers are being torn down is not in the desired state of the world".
Code change:
l.migrationCheck.validateMigrationVolumeOpCounts() } <del> ginkgo.It("should write files of various sizes, verify size, validate content [Slow][LinuxOnly]", func() { </del> <ins> ginkgo.It("should write files of various sizes, verify size, validate content [Slow]", func() { </ins> init() defer cleanup()

Related discussion:
text: After a restart, we would definitely not add a pod back that was in terminating state into desired state of the world, but "restart + this change" should be roughly equivalent to "restart and force delete while restart is processing" The other change was in which changed a live CRI read (which was trying to enforce that the dsowp is synchronized with the pod worker) into a threadsafe check against the pod worker directly (the pod worker can provide authoritative answers to "this pod has no running containers" now). The latter change seems less likely to impact you.
Thank you for the details! I think your change is making the right change, but unfortunately some of volume management still have issues, and so the tests rely on some behaviors that is not very right (pod volume is back to desired state even after it is in terminating) The cleanup mount currently does not have the logic to clean up global mount point if the volume plugin sets it up. I am still testing more and will share more details.
is this a release blocker?
Yes, this impacts volume cleanup if kubelet restarts for any reason.
On a kube-up cluster running the latest master against the latest e2e binary, I was able to reproduce the in-tree failures but not the CSI ones for some reason. Will focus on in-tree failures for the time being
/triage accepted
marking release-blocker per note on slack:
/priority critical-urgent (once upon a time we used to not ship a release if any priority/critical-urgent issues were open, not sure if we decided to stop doing that)
We were able to find the trigger for the CSI failures - on 6/29 there was a change in the hostpath driver test manifest, and the driver's is enabled. This makes the driver return an error instead of only logging a warning message on if the volume isn't unstaged properly. IIUC there's a long standing issue that reconstruction doesn't handle node unstage. could you confirm? If that's the case, then these CSI failures are not release blocking. Furthermore, the CSI test with graceful delete started to fail on 7/9 because of since graceful and force deletes are now considered the same from the point of view of the volume manager. cc
When that check was enabled, I don't think we realized it would impact the reconstruction path; it was enabled to validate a different issue.
Code change:
l.migrationCheck.validateMigrationVolumeOpCounts() } <del> ginkgo.It("should write files of various sizes, verify size, validate content [Slow][LinuxOnly]", func() { </del> <ins> ginkgo.It("should write files of various sizes, verify size, validate content [Slow]", func() { </ins> init() defer cleanup()

Related discussion:
text: Are we running into the issue that reconstruction of staging directories isn't supported () OR is it that our reconstruction path requires drivers to have real mount points ()? Either way since both are known gaps with reconstruction, I think it makes sense to to disable the flag until we can address those issues.
I was looking at this today too, trying to understand the underlying cause. That is why the failures are being reported now, but the part that I didn't understand is why NodeUnstageVolume is not being called for these volumes. In this instance: The DeleteVolume request fails because a staged path is still present: And in the kubelet log, NodeUnpublishVolume is called Then we hit this error: And NodeUnstageVolume is never called for this volume (same for other failures I looked at) so DeleteVolume keeps failing. So I think you're right that it's triggered by reconstruction, and it looks similar to , except in this case the restart happened before NodeUnpublishVolume.
And it's probably not since the test uses a PVC
Will look deeper into why UnmountDevice wasn't issued as well, but some update on the in-tree tests: the force delete test is skipped for the in-tree hostpath CSI driver because of Graceful tests are now failing as well because they behave like force tests due to
I don't think the issue is related to PVC or not. I think it's related to drivers that do not actually mount devices as a separate filesystem than the root.
There are three possible flows during kubelet restart pod gracefully delete case, the pod still exists (it is not considered as terminated in function isPodTerminated(); note that this logic has some changes after , see , so pod volume will be back to desired state, and reconciler will try to set up volume and fill in actual state. When pod volume is finally deleted from desired state, reconciler will trigger teardown with information from actual state to unpublish and unstage volume. pod is forcefully deleted, pod volume will not be in desired state. If volume reconstruction succeeds (be able to reconstruct volume spec), volume will be back into actual state. Then reconciler will trigger teardown with information from actual state to unpublish and unstage volume (this part is similar to case 1).
Code change:
l.migrationCheck.validateMigrationVolumeOpCounts() } <del> ginkgo.It("should write files of various sizes, verify size, validate content [Slow][LinuxOnly]", func() { </del> <ins> ginkgo.It("should write files of various sizes, verify size, validate content [Slow]", func() { </ins> init() defer cleanup()

Related discussion:
text: last case is reconstruction failed (in this csi-host-path test, reconstruction will fail due to issue related to , it fail to check mount point for csi hostpath driver because it is using a bind mount of the same device ). In this case, reconciler will try to clean up mounts directly, but only published volume path, not the staged volume path (this logic is not implemented due to missing information). As a workaround, we can now disable check-volume-lifecycle if it does not cause other issues. Then we will work on to make sure csi-host-path driver can pass volume mountpoint check during reconstruction.
This issue is now mitigated with merged. There are two longer-term issues that are isolated to hostpath plugins (and maybe other plugins that are similar in behavior, perhaps emptydir and similar?): related to IsLikelyNotMountPoint as mentioned. Because these don't impact the core volume machinery and are not regressions, this issue doesn't need to be release blocking. We'll close this issue once (2) is tracked.
is the IsLikelyNotMountPoint issue
after is merged, two csi-hostpath driver subpath test can pass, and two in-tree driver gracefully delete tests were skipped. https://k8s-
can you please cherry-pick to release-1.22 branch?
Opened cherry-pick, please review:
- what is the plan for reenabling them?
cherry-pick merged!
Cheng and I will work on solving IsLikelyNotMountPoint issue for hostpath drivers. That should allow test to pass and we can enable them.
Can this be removed from the milestone now that the workarounds have been merged, while leaving this issue open to track further work?
We can use existing issue to track the real fix. /close
Closing this issue. <detailsIn response to : We can use existing issue to track the real fix. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Code change:
} }) <del> ginkgo.It("should function for pod-Service(hostNetwork): udp", func() { config := e2enetwork.NewNetworkingTestConfig(f, e2enetwork.EndpointsUseHostNetwork) ginkgo.By(fmt.Sprintf("dialing(udp) %v --> %v:%v (config.clusterIP)", config.TestContainerPod.Name, config.ClusterIP, e2enetwork.ClusterUDPPort)) err := config.DialFromTestContainer("udp", config.ClusterIP, e2enetwork.ClusterUDPPort, config.MaxTries, 0, config.EndpointHostnames()) if err != nil { framework.Failf("failed dialing endpoint, %v", err) } ginkgo.By(fmt.Sprintf("dialing(udp) %v --> %v:%v (nodeIP)", config.TestContainerPod.Name, config.NodeIP, config.NodeUDPPort)) err = config.DialFromTestContainer("udp", config.NodeIP, config.NodeUDPPort, config.MaxTries, 0, config.EndpointHostnames()) if err != nil { framework.Failf("failed dialing endpoint, %v", err) } }) </del> // if the endpoints pods use hostNetwork, several tests can't run in parallel // because the pods will try to acquire the same port in the host. // We run the test in serial, to avoid port conflicts.

Related discussion:
text: <!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --ci-kubernetes-e2e-gce-cos-k8sbeta-default Networking Granular Checks: Services should function for service endpoints using hostNetwork Triage looks like it's mostly flaking on this job only: Examples /sig network <!-- Please see the deflaking doc () for more guidance! --
/milestone v1.21 (marking in the current milestone until we get confirmation whether it is relevant to the release or not)
Slack Thread:
/assign
This is fixed by , let's wait some runs to confirm before closing
/milestone clear (since this has been deemed to be non release-blocking)
/triage accepted
/close
Closing this issue. <detailsIn response to : This is fixed by , let's wait some runs to confirm before closing Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Code change:
} else { handler = restfulGetResource(getter, exporter, reqScope) } <del> handler = metrics.InstrumentRouteFunc(action.Verb, resource, subresource, namespaceScope, handler) </del> <ins> if needOverride { // need change the reported verb handler = metrics.InstrumentRouteFunc(verbOverrider.OverrideMetricsVerb(action.Verb), resource, subresource, namespaceScope, handler) } else { handler = metrics.InstrumentRouteFunc(action.Verb, resource, subresource, namespaceScope, handler) } </ins> if a.enableAPIResponseCompression { handler = genericfilters.RestfulWithCompression(handler, a.group.Context) }

Related discussion:
text: Is this a BUG REPORT or FEATURE REQUEST?: What happened: use kubectl search log with follow option get api latency (apiserverrequestlatenciessummary{quantile="0.99",resource="pods",verb="GET"} )high to give a alert in my cluster. the apiserver log is below and we can see it costs 1m25s. What you expected to happen: do not make this api latency account in getting pod request api latency How to reproduce it (as minimally and precisely as possible): use kubectl to search a pod log with follow option long enough and you will get this metric apiserverrequestlatenciessummary{quantile="0.99",resource="pods",verb="GET"} higher than the ordinary case. Anything else we need to know?: I am happy to fix it. Environment: Kubernetes version (use ): 1.6 Cloud provider or hardware configuration**: bare-metal OS (e.g. from /etc/os-release): ubuntu 16.04 Kernel (e.g. ): 4.4 Install tools: kubeadm Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig API Machinery
/area apiserver
/sig api-machinery
/assign Thanks... looking forward to your PR!
fixed with PTAL
Code change:
case "get": // Return an existing (matching) node on get. return &api.Node{ <del> ObjectMeta: api.ObjectMeta{Name: "127.0.0.1"}, Spec:       api.NodeSpec{ExternalID: "127.0.0.1"}, </del> <ins> ObjectMeta: api.ObjectMeta{Name: testKubeletHostname}, Spec:       api.NodeSpec{ExternalID: testKubeletHostname}, </ins> }, nil default: return nil, fmt.Errorf("no reaction implemented for %s", action.Action)

Related discussion:
text: Looks like some kubelet unit tests perform a lookupIp which ends up depending on the health of the local resolver. Specifically has this failing: Which leads to the test in: And we recently swapped out the node names from testnode to 127.0.0.1, so probably the culprit?
/ref
Interesting. I did swap out the node names in the unit test as part of to be 127.0.0.1, party because I thought that it wouldn't have any problems resolving during unit tests.
/cc
Code change:
case "get": // Return an existing (matching) node on get. return &api.Node{ <del> ObjectMeta: api.ObjectMeta{Name: "127.0.0.1"}, Spec:       api.NodeSpec{ExternalID: "127.0.0.1"}, </del> <ins> ObjectMeta: api.ObjectMeta{Name: testKubeletHostname}, Spec:       api.NodeSpec{ExternalID: testKubeletHostname}, </ins> }, nil default: return nil, fmt.Errorf("no reaction implemented for %s", action.Action)

Related discussion:
text: cc
What is the environment where you are running the tests? They pass for me on my mac, a linux box, and shippable seems to be passing them as well.
I couldn 't reproduce the failure either.
I'm out right now; i'll try them when i get back. For the record, I run hack/test- Paul Morie wrote:
Ahem FTR This acronym is going to go quad-plat Paul Morie wrote:
I ran on both platforms and saw the tests pass.
I'm still seeing it. Won't have bandwidth to dig in personally until tomorrow.
seeing a similar failure after a git pull
I knew I wasn't crazy
environment: fedora 21, golang: 1.4, docker: 1.6
I think I see the problem. When I set the node name to in the test I was assuming it would parse as a valid ip address and not end up at the LookupIP call. But only the node name (and not the kubelet hostname) is being set (except in my new test where I ) so it's trying to do a lookup. Do you think you could test just changing the default hostname from "testnode" to "127.0.0.1" and see if that fixes it for you?
Trying now
I still get failures but no panic / threaddump.
gist coming
this looks like again?
Good that it isn't panicing any longer. I can dig into the test failures tomorrow (I think it should be relatively simple to resolve).
cool, thanks for looking
Passes in master now for me. Closing.
FYI, they will break again once goes in, and then we can merge my PR to see if that fixes it for you.
:+1:
is in -- are these tests breaking again?
I just rebased which I think will fix them.
They are breaking again.
is on it.
Code change:
function save-logs() { local -r node_name="${1}" local -r dir="${2}" <del> local files="${3} ${common_logfiles}" </del> <ins> local files="${3}" </ins> if [[ "${KUBERNETES_PROVIDER}" == "gce" || "${KUBERNETES_PROVIDER}" == "gke" ]]; then files="${files} ${gce_logfiles}" fi if [[ "${KUBERNETES_PROVIDER}" == "aws" ]]; then files="${files} ${aws_logfiles}" fi if ssh-to-node "${node_name}" "sudo systemctl status kubelet.service" &> /dev/null; then ssh-to-node "${node_name}" "sudo journalctl --output=cat -u kubelet.service" > "${dir}/kubelet.log" || true ssh-to-node "${node_name}" "sudo journalctl --output=cat -u docker.service" > "${dir}/docker.log" || true <ins> ssh-to-node "${node_name}" "sudo journalctl --output=cat -k" > "${dir}/kern.log" || true </ins> else <del> files="${files} ${initd_logfiles} ${supervisord_logfiles}" </del> <ins> files="${kern_logfile} ${files} ${initd_logfiles} ${supervisord_logfiles}" </ins> fi copy-logs-from-node "${node_name}" "${dir}" "${files}" }

Related discussion:
text: Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): "LRUExpireCache" Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT What happened: requires write lock since needs to manipulate its list to track recently used item. Currently it uses read lock so it may introduce race condition. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else do we need to know:
This was fixed in
Code change:
}, &api.Pod{ ObjectMeta: api.ObjectMeta{ <del> UID:       "222", Name:      "bar" + "-" + hostname, Namespace: "default", SelfLink: getSelfLink("bar-"+hostname, kubetypes.NamespaceDefault), </del> <ins> UID:         "222", Name:        "bar" + "-" + hostname, Namespace:   "default", Annotations: map[string]string{kubetypes.ConfigHashAnnotationKey: "222"}, SelfLink:    getSelfLink("bar-"+hostname, kubetypes.NamespaceDefault), </ins> }, Spec: api.PodSpec{ NodeName:                      hostname,

Related discussion:
text: See for more context. Currently, kubelet compares the spec of a mirror pod (created via apiserver) with that of a static pod (config stored as a file) to decide whether the two are considered identical. If they differ, kubelet would delete the mirror pod and re-create it with the correct spec. Due to the fact that apiserver may set default values for some fields in the spec of the mirror pod, the specs could deviate undesirably. To avoid problems like this, we can store the config of a static pod as an annotation in the mirror pod, and compare the static pod with the annotation.
Would a hash of the config be sufficient if we're only going to compare? The whole spec would generally be more useful, but I wonder if that is too much data to put in an annotation.
This seems totally reasonable to me, but I'm still desperately curious why you can't just bypass defaulting when you generate mirror pod from static pod? (i.e. just use the static pod's configuration verbatim)
It's not kubelet who applies the default values here. The admission plugin was only run at the apiserver, and to bypass that, you have to ask apiserver to treat mirror pods differently and potentially at multiple places.
On May 22, 2015 10:38 AM, "Yu-Ju Hong" wrote: Understood, but is it really multiple places? If so then I agree it's a bad idea. Otherwise it might be worth considering as an alternative.
It's untenable for Kubelet to assume to know/circumvent all fields that will be set by other entities: admission control, defaulting, late initialization, auto-something, whatever. It needs to obey the same rules as other clients.
I don't understand why you would ever want to apply any of the things you mentioned to static pods. Static pods should be copied verbatim into the mirror pods, right? That's what I'm proposing.
What if the master is updated before the kubelet, and the master adds a new API field with a default value?
That case should be discussed in
I agree that mirror pod should reflect the true status of the static pod running on the node, and no default values should be set in this case.