jiebi/CodeConvo huggingface dataset:
fails in build environments without network connection due to missing in-tree vendored dependency . Build to succeed with without needing to download any go dependencies. In a network-less build environment, run . No response Building off of the v1.28.0-rc.0 tag N/A <details</detailsN/A N/A N/A
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig release
This is a hack/tools dependency here by , he may have more context
hack/tools aren't vendored. if you want to ensure those are available offline, will do that
My hack for building with OBS (), see
Is this actually a bug?
in I agree we probably don't want to calls in the build path... in code generation or verify tools, that's more ok, but the build target of a particular SHA should be able to execute hermetically
opened to scope down when we do this to test targets, and avoid the install if the env already explicitly set GOMAXPROCS
fix merged to master and head of release-1.28 branch
Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: when i create PersistentVolume, yaml like that: than describe it, got : What you expected to happen: Like that: How to reproduce it (as minimally and precisely as possible): What's said in Anything else we need to know?: Environment: Kubernetes version (use ): master Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig storage
Already got PR to fix this. /assign
master-blocking: integration-master- This has been occasionally flaking for some time. Recent Failures: Another leak was fixed at /sig api-machinery /sig testing
/retitle [Flaking Test] integration-master (goroutine leak detection)
Some server is not stopping gracefully on that test
Similar reason:
Its either not terminating gracefully or there is a request that isn't being being served in a timely manner, because I also see a bunch of goroutines waiting on the netpoller.
/triage accepted (for SIG api-machinery)
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /triage accepted (for SIG api-machinery) Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I don't think this would be a release blocker, still can you confirm whether this would block the upcoming alpha 1 release cut?
Another failure:
Hey , The deadline for the alpha 1 release cut has been delayed to Monday, June 10 and we are still waiting for confirmation of this issue. Do you think this will be a release blocker?
/assign
/assign
integration test flake, we believe this PR fixes it: I'm not marking this fixed because it also mentions a integration test flake.
The e2e test should be relaxed. It currently checks 3 different aspects of the volume mount flag: with do not receive mounts from outside their own mount namespace, nor do mount namespaces outside the container receive mounts created within the container. with receive new mounts from the host OS, the container runtime, and other containers with , but no mount namespaces outside the container receive mounts created within the container. with receive new mounts identical to containers with , but additionally any mounts made by these containers are: propagated to the container runtime where they can be propagated to other containers (related to 2) also propagated to the Host OS The first 3 aspects, up to and including 3.i, are critical for container operation and should still be ensured by an e2e test. This last check, 3.ii., is the part that I believe is overly strict. The current e2e test is too opinionated on the underlying node OS configuration. The requirement of propagation to the host OS precludes administrators or solution providers from modifying the Kubernetes installation to hide these Kubernetes-specific mountpoints, which can be for a variety of reasons: While access to container-specific mounts in Kubernetes (like secrets) today are protected by filesystem permissions, the fact that the locations are visible to unprivileged users via the 'mount' command can provide information which could be used in conjunction with other vulnerabilities to decipher secrets or other protected container mount contents. Locking access to even listing the mountpoints behind the ability to enter the mount namespace (which in turn requires and ) removes one level of potential access to container-specific mounts. These container overlay mounts, secret and configmap mounts, and namespace mounts are all internal implementation details around the way the Kubernetes and the Container Runtime allocate resources to containers. For normal day-to-day operation, there is no need for these special-use mounts to be visible in the host OS's top-level 'mount' or 'findmnt' output. After much investigation and conversation on both the sig-node and sig-storage mailing lists, as well as within Red Hat's OpenShift communities, we have yet to turn up a use case that relies on this container-to-host propagation. In fact, it may be seen as an anti-pattern and run contrary to the container-native way of doing things. Relying on a host OS to interact with a container-owned mountpoint seems suspect.
Today systemd's CPU usage is adversely affected by a large number of mountpoints, especially versions released prior to December 2020. This feature of systemd, which scans all /proc/1/mount to invoke udev events for every change suffers from two issues. The first was an inefficiency in systemd’s processing logic. This was fixed in December 2020, and this fix reportedly reduced the systemd mount-scanning-related overhead by 30-50%. The second is a lack of granularity in the events the kernel sends to systemd about mountpoint changes, which necessitates a full rescan of all mountpoints even on a minor change or single mountpoint addition. The kernel change to remedy this seems unlikely to move forward in the near term. (See for more details) For example, a fairly small OpenShift deployment today (which does not yet have the systemd December 2020 fix) has upwards of 400 mountpoints on every node owned by Kubelet, the Container Runtime, and infrastructure containers like OVN or CSI operators. Hiding the container mountpoints reduces the steady-state systemd CPU utilization from ~10% to ~0% on every worker node. On a more heavily loaded system, with 1000’s of container mountpoints, systemd’s usage will cap out at 100% of a single CPU core. Even with the 30-50% reduction in systemd usage expected from the December 2020 fix, having systemd take up as much as 50-70% of a single CPU in the worst-case scenario is a lot of unnecessary overhead. Administrators of other OSes with other host OS services that monitor or process mountpoints indiscriminately would be able to hide these container-specific mountpoints from the OS at large, at their option. With this requirement relaxed, it would be possible to run multiple instances of Kubernetes on a single host without the various mountpoints interfering with one another. There may be other inter-instance difficulties besides the mount namespace, but this does move in that direction without resorting to more complicated solutions like trying to run Kubernetes inside of containers.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available .
If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig architecture storage /area conformance
I have a proof-of-concept of the relaxed test as well as some more and an OpenShift-based proof-of-concept of an actual implementation with the container mounts in their own namespace
/sig node
There's still a number of other issues, no? Is this a real motivation / use-case? There are other difficulties. Most of the difficulties with running Kubernetes inside containers are just the difficulties in sharing a kernel / host OS, which aren't related to this at all. The security and performance angles seem relevant, in any case.
Update: I mistakenly referred to this as a "conformance test" - Turns out it is not in fact a conformance test; just an e2e test :) Edited the original post to be more correct in this respect.
/remove-area conformance /area test
Fair point. I this (intentionally vague) section since some conversation on the sig-node mailing list with folks working on KIND suggested this may align with their goals in some way I'm not entirely clear on.
A good next step on this issue is to open a PR to the related e2e tests that documents why the tests are not good candidates for future inclusion in conformance. Is that something you can assist with ?
I have a branch that relaxes the test so it works in the current environment and a separated mount namespace environment... Can you explain a bit more what you mean about documenting this? Where would that kind of documentation go?
https://k8s-
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
Most likely caused by the fluentd update /cc
The gce-gpu tests are failing for the same reason.
Ditto for pull-federation and federation-ci. pull-federation will need a cluster rebuild once the fix is incorporated upstream. /cc csbell
well that's exciting. cc
to revert fluentd to 2.0.5.
I don't understand what's special about these clusters, though. The PR passed presubmit tests a number of times.
none of the presubmit tests run COS images. I think that's the problem. From the error message, it seems like copying libsystemd is not enough, and more libraries need to be copied (e.g., libcap). Will need to build a new fluentd image to fix this.
yeah, I think you're right. I started a cluster myself and the pod is failing on the master (COS) but not on the nodes (CVM). curious that this broke between fluentd 2.0.5 and 2.0.6, which has no code changes and only updates the base image... I guess something materially changed in the base image.
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: Pods stuck in terminating state on Windows node after deleting it What you expected to happen: Pods get terminated and should disappear from kubectl How to reproduce it (as minimally and precisely as possible): Run a pod with windows container Delete it Pods are still terminating Anything else we need to know?: Could you pls help to resolve this issue. K8s pods stuck as Terminating after getting deleted on windows node with windows container(IIS). Docker ps -a not showing this container, so it is completely deleted, but in kubectl it is marked as Terminating. For networking we using OVN. kubectl logs: kubelet logs: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: Hyper-V VMs OS (e.g. from /etc/os-release): Windows Node: Master Node: NAME="Ubuntu" VERSION="16.04.4 LTS (Xenial Xerus)" ID=ubuntu IDLIKE=debian PRETTYNAME="Ubuntu 16.04.4 LTS" VERSIONID="16.04" HOMEURL="" SUPPORTURL="" BUGREPORTURL="" VERSIONCODENAME=xenial UBUNTUCODENAME=xenial Kernel (e.g. ): Linux esd-kbrnts-lnx-1 4.13.0-37-generic ~16.04.1-Ubuntu SMP Wed Mar 7 16:03:28 UTC 2018 x8664 x8664 x8664 GNU/Linux Install tools: OVN 2.7 on windows OVN 2.7.2 on linux Others: Docker Version on Windows node: Server Version: 17.06.2-ee-6
/sig windows
It needs sometime for kubelet to remove the pod. Could you wait a while and check whether the Pod removed?
Hi actually I have waited for 1 day, and it is not deleting. Can delete it only using --grace-period=0 --force. But I don't want to use this manual approach). Could you pls give some advice on this? Thanks in advance!
is a reference to this issue. Checking kubelet logs will always help to troubleshoot such problem.
Actually, I have attached kubelet log in the ticket and think the main reason is Also as there any option for kubelet to automatically force delete Terminating pods after some period? And --containerized is also not solving my issue. Thanks!
Kubelet won't delete the pod from apiserver. Since it is already not exist on the node, you could force delete it by kubectl.
It is clear, so how we can resolve this error: ? I think this is the core item of our issue. Thanks.
I'm not using OVN. Have you met this problem when using OVN?
adding Bob to comment on this. i have seen this issue before, but it has been quite a while.
Hi, Guys will be very grateful if you will provide some solution for it. We have tried to add to deployment spec but it doesn't help. Thanks.
The error message of is actually a warning message. It shouldn't block the removal of pods. Could you recheck the following things: the node, check whether the pod is still existing, e.g. by the pod exists, check if there are other error or warning message in kubelet logs; or else, the pod could be force removed safely from apiserver
Hi, I had posted this info in the ticket description. not showing this container, so it is completely deleted, but in kubectl it is displayed as Terminating.
i think you may get more support and faster responses in our slack channel for sig-windows.
So it could be safely force removed from API. And agreed with slack is a more faster way to such troubleshooting issues.
The issue seems to be related to . The only platform specific implementation for PodContainerManager is the one for . There is a default implementation (), that always returns true in the Exists call. Disabling the CgroupsPerQOS option on the Windows nodes resolves the issue.
Windows doesn't support cgroups. You should always disable it.
it is clear enough that Windows does not support cgroups). Unfortunately, it's not specified in the documentation. So to disable CgroupsPerQOS you should need to add this flags to kubelet: Maybe it will help someone... Thank you all for discussion!
[x] Original approval issue: [x] two weeks soak end date: 2 July 2020 This issue is created to allow edit by According to this APIsnoop query, there are still an endpoint which is untested. /wip /hold a set of 3 Pod Templates with a static label within a Namespace. that all 3 Pod Templates with the label in the Namespace are created. the set of Namespaced Pod Templates with a label via DeleteCollection. that all Pod Templates with the label have been deleted. Discover useragents: List endpoints hit by the test: Display endpoint coverage change: If a test with these calls gets merged, test coverage will go up by 1 points This test is also created with the goal of conformance promotion.
/sig testing /sig architecture /area conformance
/area conformance
During Conformance Meeting: Is there a way to write this test so it is resource agnostic? If we are just changing the resource... can be parameterize the resource-type
/assign
GitHub didn't allow me to assign the following users: heyste. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
GitHub didn't allow me to assign the following users: heyste. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen Merge 17 July 2020
Reopened this issue. <detailsIn response to : /reopen Merge 17 July 2020 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --When testing against OpenShift CI, I discovered a panic in the kubelet: Further investigation revealed that it is possible for to panic under certain circumstances: We hit this case when (podStatus remains nil) and (nil will be passed along to ). I'm not sure why exactly that would happen but it seems possible. Don't panic! Unclear, I've been trying to trace the code/find evidence of this happening in k8s CI but haven't found a match yet. ( would be nice :smile:) /sig node /cc /priority important-soon /milestone v1.22 Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/triage accepted
/assign
The bug is that running pods should terminate and then stop, not move to terminated. So something is getting into terminated but should never be in any other work type than terminating. We'll need more debug to determine why but I suspect that an orphaned pod is still in podSyncStatus at the time it UpdatePod is called, which means it's not seen as being in Terminating. isRuntimePod forces podSyncStatus to be a certain state, but perhaps it's should resett a terminated bool. Asked elana to add a v4 min level in the test jobs that reproduced this so we can see.
has bumped, here's a v=4 log with a panic: https://gcsweb-
Elana got the version with more logs and it is indeed the case that the pod is terminating (in this case because a CSI volume hasn't been torn down).
So the bug is: worker for the pod is still running (terminating has completed, but is executing syncTerminatedPod) runs, calls , gets a stale result for a pod (thinks the pod is still running, but that's just a stale cache result) asks pod worker to terminate the pod with a runningPod worker sees that the pod is actually not terminating, but terminated, and so RunningPod is inconsistently handled (UpdatePod is not defensive enough) worker passes the kill pod request through to syncTerminatedPod which has a null pod status, causing the panic In earlier versions of we explicitly refreshed the runtime cache in , but when and I went through review we removed it and I forgot this race was possible. The race becomes more possible here because the particular workload Elana is testing involves CSI drivers that have a bad cleanup loop, so syncTerminatedPod is reaching timeout on volume unmount (3m) before finally succeeding. This means the race window is huge (invoke UpdatePod(runningPod) on an terminated but not yet cleaned up pod). I think the real bug is that UpdatePod is not defensive enough - the difference between this race condition (seeing an older runtime container) and a human somehow starting the container up again before it is terminated is pretty similar. The important difference is that the UpdatePod{... running pod } is intended to clean up orphaned pods - those the pod worker is not tracking. But we only go into terminated when we are certain the workload is stopped, so if we simply ignore until the pod goes away it might be a while before we kill that container. We absolutely need a background worker tracking that, which is best to be a pod worker, but it's a bit more work. So easy fix (remove the possibility of the very common race) is to simply have syncTerminatingPod refresh the pod cache for that one pod. A slightly less easy fix is to have UpdatePod simply ignore the runningPod request until the next housekeeping loop (which in some cases with bad csi drivers could be a few minutes or so and in the worst case might block). The most correct fix is to ensure syncTerminated pod has an opportunity to maintain its invariants (no running containers) or have that be maintained alongside. Pondering variations here of fixes.
The easy fixes will defiintely be "safe", but "slight regression in latency in a few rare cases". Serializing the pod worker caused this, but it's in theory rare.
My understanding is that we make pod worker the source of truth for pod state. In this kind of race condition, as long as the request is sent to the pod worker, and the pod UID matches, it should be able to reconcile the state. Even if we do this, it is not atomic, the race condition still exists in theory. We probably should make the pod worker be able to reconcile the state. I don't quite understand what you mean by "until the next housekeeping loop"? Since the pod is being terminated in , what problem would it cause if we just ignore the runningPod request? If the termination is still in progress, we need to wait for it to finish anyway.
Here's the most concerning scenario: is terminated and then deleted from apiserver kills the container is taking a very long time after 2, housekeeping fires and sees the stale cache, triggers a kill (proposed change) ignore the kill request (since terminated pod is running) restarts container in the pod that references a volume mount can't make progress because volume detach can't make progress because the restarted container is holding a device lock / has a file open / w/e pod worker for that pod retries forever is the minimal fix that ignores (5) but I'm thinking through how to handle 8. I don't think we should do the easy fix (refreshing the cache isn't enough because you are vulnerable to 6-8). I'm potentially considering having the pod worker invoke syncTerminatingPod in the event syncTerminatedPod exits with an error, or simply have syncTerminatedPod do it if it would return an error to address 8.
But this basically means that after , there could still be running containers. In that case, this wouldn't be the only problem, the whole assumption the refactor builds on becomes invalid? BTW, restarting container may happen to docker, but containerd doesn't support it (at least it can't be done easily), not sure about cri-o.
cri-o doesn't support restarting a container, either.
The assumption in the kubelet is there are no more containers started by the kubelet, which is indeed safe.
That doesn’t prevent bugs in CRI implementations, user error, cosmic rays, basically computers. It’s unlikely and not something we really designed for before, but the prerefactor code would survive it. Appears to have gotten past the issue as expected. I’ll look at what it would take to mitigate the possibility of zombie containers such that the pod worker converges even if volume mount can’t be completed.
In general a little defensiveness here to unreliable external components has only minor cost, and some mitigation seems reasonable.
Opened the first part of the fix in
SG. Yeah, that should be considered a rare case. However, adding some defensive cleanup also SGTM as long as it doesn't introduce too much overhead or complexity. :)