jiebi/CodeConvo huggingface dataset:
<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --ci-kubernetes-e2e-ubuntu-gce [sig-storage] Subpath Container restart should verify that container can restart successfully after configmaps modified links to appreciated links to specific failures in spyglass appreciated Triage shows that this has been flaking quite frequently: It has flaked in 5 of the last 10 runs: /sig storage /priority important-soon <!-- Please see the deflaking doc () for more guidance! --
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Noticed some more flakes on this one
Yep, looks like this is still flaking quite frequently: Going to re-ping SIG Storage.
/assign
Test is flaking maybe half of the time. ~This test is failing in gce-ubuntu-master-default - not sure if it's failing in any other dashboards, nor whether this is run anywhere else.~ edit: This is failing on Ubuntu only. Not sure if this is limited to master branch. Fix to the original issue was in since 1.19. This is a subpath test which a configmap a container that mounts the configmap a liveness probe and waits for the container to restart. liveness probe and waits for the container to stop restarting. The test failed at the last step - the container never stopped restarting. Timeout is 2min. The liveness probe continuously checks whether a "probeFile" exists in (an emptyDir), and if it doesn't exist, whether the configmap file contains the modified value. The probeFile essentially acts as a condition which guards the probe from checking the configmap file. The test fails the liveness probe by removing the probeFile without updating the configmap just yet. It then fixes the liveness probe by updating the configmap to the modified value.
So if the container keeps restarting, the liveness probe could've been failing continuously within 2min, due to one of the following: (? not sure about this one) Command can't be executed (for example if container can't start) configmap file was never updated. Within 2min, the pod only restarted 5 times. The liveness probe's period is 2s, initial delay is 1s, and failure threshold is 1, so a restart due to liveness probe failure is triggered every 3s. The image is busybox. The command is just , so it's unlikely for the container to have crashed. There were no liveness probe unhealthy events after the test fixes the probe, and yet it continued to restart. No more events after probe is fixed.
Could we check if the configmap value was updated for sure to rule out that scenario ? I have not tried to run an end to end test locally :) so am not sure how to debug it
Will do! I ran the test in a GKE Ubuntu cluster running 1.19.6 and couldn't reproduce the failure. I'm now trying to run a cluster on GCE with Ubuntu nodes running k8s master, but haven't been successful after trying a number of techniques. If anyone happens to know how to do this let me know; I'll keep asking around otherwise.
I got a repro on a GKE cluster backed by COS. After the test failed the container ends up in Running state eventually, and the configmap file outputs the expected modified value. The fix is to increase the test timeout. PR coming up.
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: If you create a service that reference a Basic IP in a standard load balancer cluster, the service will stuck at "Pending" state for days without detailed error from Azure response. When you describe the service, it only has following: What you expected to happen: In this case Azure responded 400 with detailed error message that the referenced IP is a basic IP and cannot be used on standard LB. I would expect the detailed error message show up as events for easier diagnostic. How to reproduce it (as minimally and precisely as possible): Create a service that reference a Basic IP in a standard load balancer cluster Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/assign /area provider/azure /sig cloud-provider
/area cloudprovider
It looks like there was a change that first appeared in Kubernetes 1.29, where in 1.29 the flag on kubelet is no longer being used to populate the Node's Addresses.Hostname value. Here is the relevant output from a for a 1.28 cluster with : And here's the output from a 1.29 cluster with the same configuration: Here is the full list of parameters in case that is helpful: This is from an IBM Cloud Kubernetes Service 1.28 and 1.29 cluster. We do not think there was any cloud-configuration-manager or any other related change that would cause this difference. Is it possible it is a side effect of ? I expected the behavior to be consistent between Kubernetes releases, since I didn't find any reference to this change in Create an IBM Cloud Kubernetes Service cluster at 1.28 and 1.29 and see the difference No response <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
yeah, it seems likely my change broke this, it should keep working as you say, let me take a look /assign
/sig network /sig cloud-provider
/assign /unassign Thanks
FYI it was an unintended side effect of:
Do the sig cloud provider folks need to help here? We did make external cloud providers the default as of 1.29, which may have uncovered this. Let us know!
No, it's just a dumb kubelet startup bug. Well, not dumb, but easy to fix.
It seems the problem is that once kubelet sets the hostname the cloud controller manager does not override it so maybe the right behavior is only override it if is actually overridden ?
what problem?
we are discussing if the fix should be backported or not It seems you are relaying on the with type Hostname and this not clearly documented We also find out recently that the behavior of this address will be overriden if the external cloud provider configures it if the external cloud provider returns the hostname , then this problem will be solved without need to backport
(Well, to clarify my comments there: it's not well-defined between different clouds and node linux distros. If you know what cloud and distro you are on, then the behavior might be consistent and usable, but that still makes for a lousy kubernetes API.) Anyway, the question here is: was this actually breaking anything major, or did you just happen to notice this and think "that's weird"? (Also, if anyone else was broken by this bug, feel free to chime in; trying to figure out what to do about backporting this, because the original fix for this bug ended up introducing another bug, and the fix for that_ bug is maybe not backport-able...)
So far I have only heard from a single customer (who was using IBM Cloud Kubernetes Service) who had something in their cluster relying on cluster node's Address.Hostname value, and so when they updated to 1.29 they were broken. I believe they just modified their code to get this "Hostname" value (which was really the node IP) from somewhere else. So, to answer the question, it is somewhere between the "I noticed it and thought 'that's weird'" option and the "it broke something major" option. We have this documented in our 1.29 release notes for IBM Cloud Kubernetes Service, so other customers will have fair warning before updating. So at this point I don't consider fixing this in 1.29 as essential.
Kubernetes version (use ): kubernetes/master Environment: na What happened: What you expected to happen: How to reproduce it (as minimally and precisely as possible): see above Anything else we need to know: nope, have a fix.
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
Kubernetes version (use ): 6 Environment: GCE What happened: I created a cluster for Heapster testing purposes to debug failing test in It turned out that Summary API does not export system/misc stats in SystemContainers section (though there are kubelet and docker stats). Everything works as expected on nodes. See also https://k8s-
What node OS were you using? We don't create the system (misc) container on systemd, so it's expected that GCI nodes don't report those stats.
The default configuration for 1.4, which is: master: GCI nodes: Container VM
Are the nodes running 1.4.6 too? There was a bug with this in 1.4.5, but it should have been fixed in 1.4.6.
Yes, both are running in 1.4.6. If you want to reproduce from Heapster current head run . It will take some time and as a side effect you should have a cluster with the mentioned problem. This was catch by integration tests and I temporary disabled the check here
if you have spare cycles, can you look into this?
friendly ping
Looking at it now
I have found a couple things so far. On the containerVM nodes, cadvisor is exporting a container with "name": "/system" On the GCI Master, cadvisor is exporting a container with "name": "" The has this: "{% set system_container = "--system-cgroups=/system" -%}" (/system is the default misc container name) So the real question is: Should cadvisor export "/system" instead of "" on GCI? Or should we be configuring GCI to use "--system-"?
I thought we fix this very issue for 1.4 branch by a while back. Are you running into the same issue with 1.5?
I think I told you the wrong thing earlier. We don't support the misc (/system) cgroup on systemd systems (I can't remember why, but it was an explicit decision - or might know). The is something different, I believe a base systemd cgroup (slice). As for the misc container on ContainerVM nodes, we've been testing for the misc container in 1.5+ in the summary api test (https://k8s-) for a while, and I haven't seen this issue.
So if this is still reproducible, it's limited to 1.4. See this the bug was created for the post-fix.
Found the explanation here: The should use the flag to know the location that has the processes that are associated with , but it should not modify the cgroups of existing processes on the system during bootstrapping of the node. This is because is the on the host and it has not delegated authority to the to change how it manages .
IIUC this is intended behavior on systemd nodes. Based on the explanation provided, the kubelet doesnt have the authority to control "misc" processes on systemd.
+1. "misc" is just a bandaid for cvm. , David Ashpole <:
Ok, so I'm closing this issue.
The last 4 runs of https://k8s- have failed on the and steps with : https://k8s- https://k8s- https://k8s- https://k8s- The GCE version of this test suite appears to be fine. This is blocking the submit queue. CC
Is there a way to see what does /cluster/gce/list- output in the runs... exit status 1 is not very informative
Here is a non-gke failure: https://k8s-
Given that the last thing output from the listing of resource is: I would bet that the defined is failing to find a match from (see )
Hummm I ran it locally against but did not get the same failure
you might be more familiar with list-
This started happening on the 16th and became more regular on the 18th-- do you think the new janitor code might be causing this? More details here:
is giving me exit status 1, let me find which command it is.
Yeah, and have figured it out. Because of the janitor, we no longer have any default firewall rules in these projects (which might be a bug), and so now the is failing. I think in the distant past always printed out a header, even if nothing was found, though it hasn't done that in a long time. Fix for is to the on L52. We may also want to decide if we should be deleting the default firewalls.
We'll want to cherrypick that to the old branches, too.
hummm I'll let it skip default-firewall
Keeping open until we've verified fix
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: cAdvisor has a option to only collect docker related cgroups. Kubelet stats api now supports allocatable cgroup, /kubepods, for the runtime cgroup and the kubelet cgroup. But with settings, Kubelet stats api can not query pod related cgroups metrics. FYI: What you expected to happen: should be compatible with kubelet stats api.
/cc
/sig node /sig instrumentation
add to cadvisor in A comma-separated list of cgroup path prefix that needs to be collected even docker-only is specified bump cadvisor dep for Kubernetes setting to
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or .
/kind failing-test <!-- If this is a flake, please add: /kind flake --/kind flake <!-- Please assign a SIG using: /sig SIG-NAME --/sig testing apps
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
can you PTAL, it seems this test was in It doesn't seem to have a high rate of failure , but seems flakes are legit
/assign
side note: it's really hard to make it fail in a fast machine :) I had to temporarily add a sleep in the reconciler.
have you tried Jordan's instructions compiling and running with ? for integration you should have an external etcd instance
I haven't. Good to know for next time :)
Update the godoc on the encryption config API on how to specify group/resources to be encrypted ref: godoc has the details na No response_ na na <details</details<details</details<details</details<details</details
/sig auth /triage accepted
/assign Cool-Coder174
Thanks for volunteering for this LMK if you need any help with it.
Just for clarification, I just need to add documentation on how to configure encryption on groups/resources?
You can update godoc to mention that custom resources can now be to the list of to be encrypted as well and how to add the of the resource to the list. e.g. from this test
/assign /unassign Release v1.26 deadlines are here so we need to get this completed ASAP.
/milestone v1.26
I apologize for the slow actions. I know time and contributions are valuable. May I request a walk through. This is my first time ever contributing and I really want to learn. I have been reading some of the code base on my spare time.
this feature is on the harder side of things to get started with. Reading the blog post and the associated documentation is probably the best starting point:
& what is the status on this issue? 1.26 is coming up! cc
/milestone v1.27
tests are failing atm on GCI https://k8s- because VOLUMEPLUGINDIR=/home/kubernetes/flexvolume set in takes precedence over KUBELETTESTVOLUMEPLUGINDIR="/etc/srv/kubernetes/kubelet-plugins/volume/exec". As far as I can tell /home/kubernetes/flexvolume is not readable yet by containerized kubelet so the test needs to keep using "/etc/srv/kubernetes/kubelet-plugins/volume/exec" and the bash needs to be shuffled around to solve this. cc
/sig storage
This is more an acknowledgment of the issue than anything, can't write a PR until later
cc /kind e2e-test-failure
this is currently blocking alpha.3 release
we are just waiting for approval
I think they’re still failing, not sure what the release process is, but these tests started running only yesterday and so failure indicates something wrong with the test themselves, they shouldn’t block release. (They succeed on gce cluster with Debian nodes but fail on gci)
well, let's see if we can get the kinks worked out before the beta release on 9/27 or we will need to disable it from release-blocking
/assign /reopen
I0824 17:49:28. 8562 ] operationExecutor.VerifyControllerAttachedVolume started for volume "flex-volume-0" (UniqueName: "flexvolume-k8s/dummy-e2e-tests-flexvolume-wf48v/-88f4-11e7-a3f9--flex-volume-0") pod "flex-client" (UID: "-88f4-11e7-a3f9-") edit: VerifyControllerAttachedVolume must succeed since the plugin is non-attachalbe and no error is reported. But the voluem is absent from node.volumesInUse I0824 17:44:06. 4 ] AttachVolume.Attach succeeded for volume "flex-volume-0" (UniqueName: "flexvolume-k8s/dummy-attachable-e2e-tests-flexvolume-j8962/flex-volume-0") from node "bootstrap-e2e-minion-group-9913" W0824 17:44:06.
4 plugin-] flexVolume driver k8s/dummy-attachable-e2e-tests-flexvolume-j8962: using default GetVolumeName for volume 0x1c5df60 I0824 17:43:59. 3728 ] operationExecutor.VerifyControllerAttachedVolume started for volume "flex-volume-0" (UniqueName: "flexvolume-k8s/dummy-attachable-e2e-tests-flexvolume-j8962/flex-volume-0") pod "flex-client" (UID: "-88f3-11e7-a3f9-") E0824 17:43:59. 3728 ] Operation for ""flexvolume-k8s/dummy-attachable-e2e-tests-flexvolume-j8962/flex-volume-0"" failed. No retries permitted until 2017-08-24 17:44:00. +0000 UTC (durationBeforeRetry 500ms). Error: Volume has not been to the list of VolumesInUse in the node's volume status for volume "flex-volume-0" (UniqueName: "flexvolume-k8s/dummy-attachable-e2e-tests-flexvolume-j8962/flex-volume-0") pod "flex-client" (UID: "-88f3-11e7-a3f9-") Seems there is something wrong with VerifyControllerAttachedVolume, still at a loss as to why it's showing up in GCI but not Debian edit: get node -o yaml shows: yet it errors because the volume hasn't been to volumesInUse??? Aug 24 20:31:42 mawong-e2e-minion-group-npjh kubelet[9431]: E0824 20:31:42. 9431 ] Operation for ""flexvolume-k8s/dummy-attachable-e2e-tests-flexvolume-z8xc4/flex-volume-0"" failed. No retries permitted until 2017-08-24 20:31:44. +0000 UTC (durationBeforeRetry 2s).
Error: Volume has not been to the list of VolumesInUse in the node's volume status for volume "flex-volume-0" (UniqueName: "flexvolume-k8s/dummy-attachable-e2e-tests-flexvolume-z8xc4/flex-volume-0") pod "flex-client" (UID: "-890b-11e7-9392-")
Never mind the GCI vs Debian issue, our test fix would have worked but in between and , something else changed and is causing both GCI and Debian tests to fail.
might be the cause, so we might be looking at legitimate failures caught by the test at this point (yay?) and not faulty tests. investigating...
It seems that exec probes must emit "ok" to be considered success, rather than return 0. If this is what we really want, we should document it somewhere.
This is (was?) a limitation of docker exec, which didn't return the code. Brendan On Jan 28, 2015 8:38 AM, "Tim Hockin" wrote:
The doc says it looks at exit status, so we need to fix one or the other.
for kubelet
for exec
could you take a look at this? Feel free to re-assign it back to me.
FWIW the current behavior isn't SO bad, and it makes clear a failure in the thing that probed vs a failed probe. Just saying, we don't NECESSARILY need to change it. On Jan 28, 2015 5:26 PM, "Dawn Chen" wrote:
With docker 1.5 we can go back to using the exitCode instead of "ok" string. OOM kills will also have special exitCodes in docker 1.5. , Tim Hockin wrote:
ok, I'm going to change the docs for now, we can change back when we standardize on 1.5...
Nodes Network when a minion node becomes unreachable [replication controller] recreates pods scheduled on the unreachable minion node AND allows scheduling of pods on a minion after it rejoins the cluster(Failed 17 times in the last 30 runs. Stability: 43 %)Failed with 0 retries
for triage
I'll look into this.
I talked to about this and he had a pretty good guess in about two seconds about what the problem is. ReplicationController sync method does this: but DaemonSet controller doesn't. (I guess when DaemonSet was copy-pasted from ReplicationController, this was accidentally removed.) Pod reflector and syncer are running in separate goroutines, and if pod reflector doesn't run then syncer will think there are no pods on the node.
To be clear: this is about the test Controller Manager should not create/delete replicas across restart which is the one that is actually failing.
I'm confused you're saying that this is about , but you're updating . I thought they are about two different tests...
Yes sorry, I put my comment in the wrong issue. Reassigning this one to you, and I'll copy my comment into the other one.
First failure: We check if all namespace are deleted. If other tests failed to delete their namespace this test will fail. I'm going to delete this logic as we already check if we have successfully deleted our namespace. See also for some more details.
Second failure: We disable network by running the following command: I checked and it seems that it results in ing master to hang. I suspect that Kubelet hangs on updating node status and depending when it breaks the connection (by timeout) it will fit in test timeouts or not.
What is the default timeout for connections using client libraries to apiserver? I'm checking the code, but can't find it.
I'm not sure if we set one, but we should. I guess I would expect to see that in the config, w/ default set by this function: , Filip Grzadkowski wrote:
So: let's add a timeout, and revert .
I think that adding a timeout and reverting are orthogonal. I we want to simulate network problems with node-master communication then using REJECT is better anyway. At the same time adding timeout makes sense for many other reasons.