jiebi/CodeConvo huggingface dataset:
Code change:
} available := false for _, pod := range daemonPods { <del> if podutil.IsPodAvailable(pod, ds.Spec.MinReadySeconds, metav1.Now()) { </del> <ins> //for the purposes of update we ensure that the Pod is both available and not terminating if podutil.IsPodAvailable(pod, ds.Spec.MinReadySeconds, metav1.Now()) && pod.DeletionTimestamp == nil { </ins> available = true break }

Related discussion:
text: <!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: The DaemonSet controller will schedule a pod on a node when the old pod is still in Terminating state. This breaks the expectation that only one pod will be running at a time per node. What you expected to happen: The DaemonSet controller should not let two pods running at the same time on the same node. How to reproduce it (as minimally and precisely as possible): Create a DaemonSet. Set terminationGracePeriodSeconds to be relatively long. Kill a pod. Watch a new pod start on the same node when the old pod is still terminating. Anything else we need to know?: Seems this behavior was introduced in Environment: Kubernetes version (use ): 1.7.2 Cloud provider or hardware configuration**: GKE OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig apps
/cc
cc
cc: I think the expected behavior is correct, the terminating pod could be performing an important step such as 'removing a lock' or 'cleanup' etc., we should allow that to gracefully die before creating a new one. IMHO we should revoke to old behavior, but why did we need this new behavior in the first place? That is not clear to me from the PR and Issue.
We don't currently guarantee at least or at most one scheduling for DaemonSet. Both behaviors are under consideration via issue and in the future work section of the . The reason for is to "Make sure that controllers are not adopting children that are marked for deletion" (). This is necessary for GC to work properly.
Well the behavior may be technically undefined, but can I request that in the future changes like this be mentioned in the release notes?
Yes, certainly.
Code change:
} available := false for _, pod := range daemonPods { <del> if podutil.IsPodAvailable(pod, ds.Spec.MinReadySeconds, metav1.Now()) { </del> <ins> //for the purposes of update we ensure that the Pod is both available and not terminating if podutil.IsPodAvailable(pod, ds.Spec.MinReadySeconds, metav1.Now()) && pod.DeletionTimestamp == nil { </ins> available = true break }

Related discussion:
text: Also "The DaemonSet controller should not let two pods running at the same time on the same node." is not necessarily the same thing as respecting termination prior to launching a new Pod. The former implies a more strict requirement than the latter. I think we should consider at least attempting to wait for a terminating Pod to be deleted prior to creating another Pod on the Node for same DaemonSet as default behavior.
will something like work then?
Code change:
func (c *csiMountMgr) SetUpAt(dir string, mounterArgs volume.MounterArgs) error { klog.V(4).Infof(log("Mounter.SetUpAt(%s)", dir)) <del> corruptedDir := false mounted, err := isDirMounted(c.plugin, dir) if err != nil { if isCorruptedDir(dir) { corruptedDir = true // leave to CSI driver to handle corrupted mount klog.Warning(log("mounter.SetUpAt detected corrupted mount for dir [%s]", dir)) } else { return errors.New(log("mounter.SetUpAt failed while checking mount status for dir [%s]: %v", dir, err)) } } if mounted && !corruptedDir { klog.V(4).Info(log("mounter.SetUpAt skipping mount, dir already mounted [%s]", dir)) return nil } </del> csi, err := c.csiClientGetter.Get() if err != nil { return volumetypes.NewTransientOperationFailure(log("mounter.SetUpAt failed to get CSI client: %v", err))

Related discussion:
text: What happened: A custom CSI driver implementing returns a failure response, indicating that it did not complete its intended steps to prepare the volume for the mounting pod. Let's say that these steps involve a filesystem at the target directory, data from a service, and that data to a set of files in the mounted filesystem. If the first filesystem mounting step succeeds, but either of the subsequent two steps fail, the volume is still not ready for use by the pod. However, if this CSI driver returns a failure response, and the CSI mounter comes back and, before invoking the RPC again, it finds a filesystem already mounted at the target directory, , and ceases further calls to to allow the CSI driver to try again. What you expected to happen: If a CSI driver returns a failure response to a RPC, Kubernetes should honor that declaration of failure and invoke the RPC again, disallowing the mounting pod from starting until this RPC succeeds. Even though it's possible to partially work around the current behavior by having the driver mount the filesystem as late as possible, and then attempting to unmount it if fails to populate the volume with its intended data, doing so makes writing drivers more complicated and forces them to do extra work. How to reproduce it (as minimally and precisely as possible): In a CSI driver (even one that serves just ephemeral, inline volumes), implement to mount a filesystem at the target path, then return a failure response. Confirm that Kubernetes does not invoke again, because there's now a filesystem mounted at the target path. Anything else we need to know?: This topic came up for discussion in the "Kubernetes" Slack team. There, helped investigate this complaint, and to discuss changing this behavior. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: AWS EC2 OS (e.g: ):Kernel (e.g. ):Install tools: kubeadm Others: csi-node-driver-registrar container image tag: v1.0.2
/sig storage
While the plugin should try to clean up as much as possible, I agree that kubelet should not be assuming that if there's a mount point, the publish is successful. NodePublish is supposed to be idempotent so it should theoretically be safe to call it when it's already mounted, although there is the possibility that plugins may have written against the K8s behavior.
Code change:
func (c *csiMountMgr) SetUpAt(dir string, mounterArgs volume.MounterArgs) error { klog.V(4).Infof(log("Mounter.SetUpAt(%s)", dir)) <del> corruptedDir := false mounted, err := isDirMounted(c.plugin, dir) if err != nil { if isCorruptedDir(dir) { corruptedDir = true // leave to CSI driver to handle corrupted mount klog.Warning(log("mounter.SetUpAt detected corrupted mount for dir [%s]", dir)) } else { return errors.New(log("mounter.SetUpAt failed while checking mount status for dir [%s]: %v", dir, err)) } } if mounted && !corruptedDir { klog.V(4).Info(log("mounter.SetUpAt skipping mount, dir already mounted [%s]", dir)) return nil } </del> csi, err := c.csiClientGetter.Get() if err != nil { return volumetypes.NewTransientOperationFailure(log("mounter.SetUpAt failed to get CSI client: %v", err))

Related discussion:
text: For the mount point checking here before calling NodePublishVolume, I am wondering it is just for performance optimization, or it has other purposes. Have concern about some drivers might depend on this check if they are not implemented with idenpotency.
Those drivers would be operating in violation of the specification. I'd prefer that we don't penalize already conforming and yet-to-be written drivers in order to accommodate these existing drivers—if they even exist.
More fallout in last night's testing: I introduced a bug into my driver that caused it to panic during ("Writing that data to a set of files in the mounted filesystem"). My handler function is set up to unmount a successfully mounted volume via if it doesn't complete steps 2 and 3 above, in order to deal with this issue's principal complaint. My driver container would crash due to this panic. The kubelet would call on my driver twice, it would crash twice, but the pod still started running. I'm not yet sure whether or not the filesystem got unmounted successfully via the on the first call, but somehow the pod started despite my handler function never completing successfully. I'm going to try to get some more debugging time on this later today to see if I can figure out what the kubelet was doing there.
Studying my code more, I see that my function only unmounted the mounted directory when the containing handler function was returning an error. When unwinding due to a panic, the named return value of type I was inspecting was still nil. Revising the criteria in that function covers both failure reasons (returning an and unwinding due to a panic). Regardless, this further underscores the problem: If a driver makes it as far as mounting a filesystem at the target directory, but fails afterward—including the process crashing and never responding positively to the RPC—the mounting pod will start. That's likely to then cause the pod to fail, for lack of the files it expected to find mounted.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity.
Code change:
func (c *csiMountMgr) SetUpAt(dir string, mounterArgs volume.MounterArgs) error { klog.V(4).Infof(log("Mounter.SetUpAt(%s)", dir)) <del> corruptedDir := false mounted, err := isDirMounted(c.plugin, dir) if err != nil { if isCorruptedDir(dir) { corruptedDir = true // leave to CSI driver to handle corrupted mount klog.Warning(log("mounter.SetUpAt detected corrupted mount for dir [%s]", dir)) } else { return errors.New(log("mounter.SetUpAt failed while checking mount status for dir [%s]: %v", dir, err)) } } if mounted && !corruptedDir { klog.V(4).Info(log("mounter.SetUpAt skipping mount, dir already mounted [%s]", dir)) return nil } </del> csi, err := c.csiClientGetter.Get() if err != nil { return volumetypes.NewTransientOperationFailure(log("mounter.SetUpAt failed to get CSI client: %v", err))

Related discussion:
text: Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Code change:
Status: v1.ConditionTrue, }) <del> hostIP, err := kl.getHostIPAnyWay() if err != nil { glog.V(4).Infof("Cannot get host IP: %v", err) return *s } s.HostIP = hostIP.String() if kubecontainer.IsHostNetworkPod(pod) && s.PodIP == "" { s.PodIP = hostIP.String() </del> <ins> if kl.kubeClient != nil { hostIP, err := kl.getHostIPAnyWay() if err != nil { glog.V(4).Infof("Cannot get host IP: %v", err) } else { s.HostIP = hostIP.String() if kubecontainer.IsHostNetworkPod(pod) && s.PodIP == "" { s.PodIP = hostIP.String() } } </ins> } return *s }

Related discussion:
text: http://k8s- /priority failing-test /kind bug /status approved-for-milestone /sig cluster-lifecycle the job is on the master-blocking dashboard and the e2e test has been failing, please take a look cc
testgrid change suggests
(author of the change in that range) -- any idea whether might have broken reboot tests?
I reverted the change as it may lead to this regression. The other change is really small () and unlikely related to this. Still the continues to say otherwise, I think we can reopen this issue.
The reboot test is still failing.
Just opened to achieve the revert.
Is the test passing now? Can this issue be closed?
looks green now
Code change:
# These are "local" but are visible to and relied on by functions this # function calls.  They are effectively part of the calling API to # build_binaries_for_platform. <del> local goflags goldflags goasmflags gogcflags gotags </del> <ins> local goflags goldflags gogcflags gotags </ins> <del> # This is $(pwd) because we use run-in-gopath to build.  Once that is # excised, this can become ${KUBE_ROOT}. local trimroot # two lines to appease shellcheck SC2155 trimroot=$(pwd) goasmflags="all=-trimpath=${trimroot}" </del> <ins> goflags=() gogcflags="${GOGCFLAGS:-}" goldflags="all=$(kube::version::ldflags) ${GOLDFLAGS:-}" </ins> <del> gogcflags="all=-trimpath=${trimroot} ${GOGCFLAGS:-}" </del> if [[ "${DBG:-}" == 1 ]]; then # Debugging - disable optimizations and inlining and trimPath <del> gogcflags="${GOGCFLAGS:-} all=-N -l" goasmflags="" fi goldflags="all=$(kube::version::ldflags) ${GOLDFLAGS:-}" if [[ "${DBG:-}" != 1 ]]; then # Not debugging - disable symbols and DWARF. </del> <ins> gogcflags="${gogcflags} all=-N -l" else # Not debugging - disable symbols and DWARF, trim embedded paths </ins> goldflags="${goldflags} -s -w" <ins> goflags+=("-trimpath") </ins> fi # Extract tags if any specified in GOFLAGS

Related discussion:
text: While working on for , I found that our kubernetes1.24 package stopped to produce bit-identical results on 2 fresh builds. The first chunk is likely just the hash of the remainder, so can be ignored. The other chunks indicate that a random build temporary path is used for Sources should be able to reproduce bit-identical binaries e.g. for SLSA level4 On Debian or openSUSE do No response <detailsWARNING: This version information is deprecated and will be replaced with the output from kubectl version --short. Use --output=yaml|json to get the full version. Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2022-05-05T00:00:00Z", GoVersion:"go1.18.3", Compiler:"gc", Platform:"linux/amd64"} Kustomize Version: v4.5.4 console $ cat /etc/os-release $ uname -a Linux bmwiedemann-factory 5.18.6-1-default SMP PREEMPTDYNAMIC Thu Jun 23 05:46:18 UTC 2022 () x8664 x8664 x86_64 GNU/Linux </details<detailsgo1.18 </details<details</details<details</details
/sig release /area build-release
can you please post a link to the full logs from the build?
any news on this? Haven't heard from (context: we worked to fix this a couple of times in )
Unfortunately not, is this something we can fix on the OBS side? If the go build timestamp () is the main issue here then we can approach a fix in our build pipeline.
It would be good if would not be embedded in the first place. It might be possible to replace it somehow in our builds, but all other people wanting reproducible builds, would need to replicate that.
full logs from the build: from kubernetes-1.24.3 built 2022-08-18
can you please try setting GOCACHE=off when you run the build? I am suspecting that the temp directory is being generated here: (or add an export in for that env var)
I to but still got these diffs.
Code change:
# These are "local" but are visible to and relied on by functions this # function calls.  They are effectively part of the calling API to # build_binaries_for_platform. <del> local goflags goldflags goasmflags gogcflags gotags </del> <ins> local goflags goldflags gogcflags gotags </ins> <del> # This is $(pwd) because we use run-in-gopath to build.  Once that is # excised, this can become ${KUBE_ROOT}. local trimroot # two lines to appease shellcheck SC2155 trimroot=$(pwd) goasmflags="all=-trimpath=${trimroot}" </del> <ins> goflags=() gogcflags="${GOGCFLAGS:-}" goldflags="all=$(kube::version::ldflags) ${GOLDFLAGS:-}" </ins> <del> gogcflags="all=-trimpath=${trimroot} ${GOGCFLAGS:-}" </del> if [[ "${DBG:-}" == 1 ]]; then # Debugging - disable optimizations and inlining and trimPath <del> gogcflags="${GOGCFLAGS:-} all=-N -l" goasmflags="" fi goldflags="all=$(kube::version::ldflags) ${GOLDFLAGS:-}" if [[ "${DBG:-}" != 1 ]]; then # Not debugging - disable symbols and DWARF. </del> <ins> gogcflags="${gogcflags} all=-N -l" else # Not debugging - disable symbols and DWARF, trim embedded paths </ins> goldflags="${goldflags} -s -w" <ins> goflags+=("-trimpath") </ins> fi # Extract tags if any specified in GOFLAGS

Related discussion:
text: <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This is still relevant even with 1.27. It might be worth revisiting this issue again. /reopen /triage accepted
Reopened this issue. <detailsIn response to : This is still relevant even with 1.27. It might be worth revisiting this issue again. /reopen /triage accepted Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle rotten
latest run -
I found that our package has the same issue, so it probably it is a toolchain issue in golang - and much faster to debug outside of the slow kubernetes build. =filed
AIUI, the conclusion by the golang people seems to be that commit is not the right approach and one should use with the go call instead. Not sure how that interacts with other parts of your build.
cc (please see referenced commit above!)
I see. I inherited the gcflags before so I plead innocence. It seems plausible to change it from to - I don't know when I will have time to look at that, though.
Code change:
# These are "local" but are visible to and relied on by functions this # function calls.  They are effectively part of the calling API to # build_binaries_for_platform. <del> local goflags goldflags goasmflags gogcflags gotags </del> <ins> local goflags goldflags gogcflags gotags </ins> <del> # This is $(pwd) because we use run-in-gopath to build.  Once that is # excised, this can become ${KUBE_ROOT}. local trimroot # two lines to appease shellcheck SC2155 trimroot=$(pwd) goasmflags="all=-trimpath=${trimroot}" </del> <ins> goflags=() gogcflags="${GOGCFLAGS:-}" goldflags="all=$(kube::version::ldflags) ${GOLDFLAGS:-}" </ins> <del> gogcflags="all=-trimpath=${trimroot} ${GOGCFLAGS:-}" </del> if [[ "${DBG:-}" == 1 ]]; then # Debugging - disable optimizations and inlining and trimPath <del> gogcflags="${GOGCFLAGS:-} all=-N -l" goasmflags="" fi goldflags="all=$(kube::version::ldflags) ${GOLDFLAGS:-}" if [[ "${DBG:-}" != 1 ]]; then # Not debugging - disable symbols and DWARF. </del> <ins> gogcflags="${gogcflags} all=-N -l" else # Not debugging - disable symbols and DWARF, trim embedded paths </ins> goldflags="${goldflags} -s -w" <ins> goflags+=("-trimpath") </ins> fi # Extract tags if any specified in GOFLAGS

Related discussion:
text: My patch doesn't seem to fix the whole problem: thockin-glaptop4 kubernetes buildtrimpath /$ make kubelet +++ [1114 19:10:12] Building go targets for linux/amd64 (non-static) thockin-glaptop4 kubernetes buildtrimpath /$ md5sum output/bin/kubelet output/bin/kubelet thockin-glaptop4 kubernetes buildtrimpath /$ make kubelet +++ [1114 19:12:40] Building go targets for linux/amd64 (non-static) thockin-glaptop4 kubernetes buildtrimpath /$ md5sum output/bin/kubelet output/bin/kubelet thockin-glaptop4 kubernetes build_trimpath /$
diff --- a 2023-11-14 19:13:45. -0800 +++ b 2023-11-14 19:13:56. -0800 -1,4 +1,4 -38rZHdQckYF9EEzmndR/BAXOwzCcq0Mk1UHT62ha/QLiRgXP4oojWh3oWP9/TWG7KW90IL8vKt1UOEPR +LeLArvl7sgWAd55ROKN2/BAXOwzCcq0Mk1UHT62ha/QLiRgXP4oojWh3oWP9/cYb5ZGQwa6js077YiAvK /lib64/ld-linux-x86- AUATUSH []AA] -,8 +,8 go1.21.3 runtime/cgo: call frame too large -2023-11-15T03:12:40Z -2023-11-15T03:12:40Z +2023-11-15T03:13:47Z +2023-11-15T03:13:47Z pthreadcreate failed: %s bad stack bounds: lo=%p hi=%p expand 32-byte kexpand 32-byte k
That seems unrelated. I tested with and it made our openSUSE package fully reproducible. Your diff contains a timestamp - maybe would get rid of that. The top part of the diff is just the go build-id hash of the remainder and can be ignored.
Code change:
# These are "local" but are visible to and relied on by functions this # function calls.  They are effectively part of the calling API to # build_binaries_for_platform. <del> local goflags goldflags goasmflags gogcflags gotags </del> <ins> local goflags goldflags gogcflags gotags </ins> <del> # This is $(pwd) because we use run-in-gopath to build.  Once that is # excised, this can become ${KUBE_ROOT}. local trimroot # two lines to appease shellcheck SC2155 trimroot=$(pwd) goasmflags="all=-trimpath=${trimroot}" </del> <ins> goflags=() gogcflags="${GOGCFLAGS:-}" goldflags="all=$(kube::version::ldflags) ${GOLDFLAGS:-}" </ins> <del> gogcflags="all=-trimpath=${trimroot} ${GOGCFLAGS:-}" </del> if [[ "${DBG:-}" == 1 ]]; then # Debugging - disable optimizations and inlining and trimPath <del> gogcflags="${GOGCFLAGS:-} all=-N -l" goasmflags="" fi goldflags="all=$(kube::version::ldflags) ${GOLDFLAGS:-}" if [[ "${DBG:-}" != 1 ]]; then # Not debugging - disable symbols and DWARF. </del> <ins> gogcflags="${gogcflags} all=-N -l" else # Not debugging - disable symbols and DWARF, trim embedded paths </ins> goldflags="${goldflags} -s -w" <ins> goflags+=("-trimpath") </ins> fi # Extract tags if any specified in GOFLAGS

Related discussion:
text: Thanks for confirming
Will there be backports to 1.24++ ? I found it applies cleanly to 1.26++ and works with some small tweaks in 1.24 and 1.25. If not, I'd submit the patch for our openSUSE packages.
1.24 and 1.25 reached EOL and we don't support these branches any longer. We can cherry-pick this to 1.26+ if folks ( agree
Cherry-picks: 1.29 1.28 1.27 1.26
Code change:
Containers: []v1.Container{ { Name:    contName, <del> Image:   BusyBoxImage, </del> <ins> Image:   AgnHostImage, </ins> Command: command, }, },

Related discussion:
text: <!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: N/A Which test(s) are failing: Since when has it been failing: Ever Reason for failure: Vm on Azure with no public IP assigned aren't able to perform a command to the internet. Anything else we need to know: The test (see ) tries to test the internet connection for a pod by pinging 8.8.8.8. But azure does not provide ICMP support for VM without a public IP therefore this test cannot succeed.(see ) A fix would be to change the test method to a tcp request instead of a ping request (the test already performs a http get to in its BeforeSuite (see ))
/sig network /sig testing /sig azure
you may refer to (Pods don't have connectivity to external networks )
cc WDYT about the proposal to change this from an ICMP call? We probably need input from sig-architecture as well since this as well /sig architecture
/triage unresolved
An alternative is to issue a TCP/UDP request to 8.8.8.8 on port 53.
ICMP is dropped by Azure for security reasons. This is intended. Can use curl as a replacement. I will fix the corresponding test.
SGTM
Can you please review the related PR? I'll in the PR again. Thanks!