jiebi/CodeConvo huggingface dataset:
Code change:
klog.V(4).Info(log("attacher.Attach finished OK with VolumeAttachment object [%s]", attachID)) <ins> // TODO(71164): In 1.15, return empty devicePath </ins> return attachID, nil } <del> func (c *csiAttacher) WaitForAttach(spec *volume.Spec, attachID string, pod *v1.Pod, timeout time.Duration) (string, error) { </del> <ins> func (c *csiAttacher) WaitForAttach(spec *volume.Spec, _ string, pod *v1.Pod, timeout time.Duration) (string, error) { </ins> source, err := getCSISourceFromSpec(spec) if err != nil { klog.Error(log("attacher.WaitForAttach failed to extract CSI volume source: %v", err)) return "", err } <ins> attachID := getAttachmentName(source.VolumeHandle, source.Driver, string(c.plugin.host.GetNodeName())) </ins> skip, err := c.plugin.skipAttach(source.Driver) if err != nil { klog.Error(log("attacher.Attach failed to find if driver is attachable: %v", err))

Related discussion:
text: A workaround in CSI volume plugin won't really help here, it will work only for drivers without .
I finally reproduced it on AWS with a long sleep in : Steps to reproduce (with the hack above): a pod with a volume the pod. Wait until is called. a new pod with the same volume. - checks that the volume is still attached and the device is still mounted and enqueues an operation to mount the volume. When finishes, starts and 1) uses empty and 2) calls on volume that's not DeviceMounted.
Fix: Any idea how to test it for real? Fake CSI plugin with slow NodeUnstage would be the best, however we don't have one.
I'm going to work on a targeted fix for CSI. We can continue to pursue fixing aws in-tree separately.
PR to fix csi is here:
what are the final set of PRs pending for this? , and ?
Only is needed to fix this particular test failure.
can you please confirm if we are going to fix this using or ?
I discussed with offline, he has some suggestions for that I am investigating. I think we will want both PRs eventually, although only one of them will be needed to fix the CSI test for now. Since it's his off hours, I will continue working on today.
Are we talking 1.13 for as well?
Reopening this since we are seeing consistent failures in latest runs. We also see one more test fail in the job with same error. Failing job https://k8s- Failing Tests [sig-storage] [Serial] CSI Volumes CSI plugin test using CSI driver: gcePD [Serial] should provision storage [sig-storage] [Serial] CSI Volumes CSI plugin test using CSI driver: hostPath [Serial] should provision storage Error This is now 1.13 release blocking.
/priority critical-urgent /kind failing-test
/remove-priority important-soon
I think this is a different issue and could be due to the fact that CSI is intentionally not backwards compatible between 1.12 and 1.13. We may need to enhance the test to figure out what version the cluster is on and launch appropriate versions of the CSI driver.
"intentionally not backwards compatible" -- is there a writeup for the release notes and release announcement for this change?
Code change:
klog.V(4).Info(log("attacher.Attach finished OK with VolumeAttachment object [%s]", attachID)) <ins> // TODO(71164): In 1.15, return empty devicePath </ins> return attachID, nil } <del> func (c *csiAttacher) WaitForAttach(spec *volume.Spec, attachID string, pod *v1.Pod, timeout time.Duration) (string, error) { </del> <ins> func (c *csiAttacher) WaitForAttach(spec *volume.Spec, _ string, pod *v1.Pod, timeout time.Duration) (string, error) { </ins> source, err := getCSISourceFromSpec(spec) if err != nil { klog.Error(log("attacher.WaitForAttach failed to extract CSI volume source: %v", err)) return "", err } <ins> attachID := getAttachmentName(source.VolumeHandle, source.Driver, string(c.plugin.host.GetNodeName())) </ins> skip, err := c.plugin.skipAttach(source.Driver) if err != nil { klog.Error(log("attacher.Attach failed to find if driver is attachable: %v", err))

Related discussion:
text: Yes has the release note
thanks, I'll also call it to the Release Notes lead's attention
what does it mean from the user impact perspective for CSI to not be backward compatible with 1.12? can you please point me to any discussions on this behavior for 1.13?
Kubernetes v1.13 speaks CSI v1.0 and no longer understands CSI v0.x. Any CSI drivers deployed on previous versions of Kubernetes speaking CSI v0.x must removed, updated to be CSI 1.0 compatible, and redeployed on the on Kubernetes v1.13. If only the master is upgraded to 1.13 (and nodes are 1.12 or earlier) older drivers will continue to work. When nodes are upgraded to 1.13, then older drivers will stop working. I will capture this in detail in the "Action Required" section.
Im tracking these upgrade test failures elsewhere: They are not related to this original issue, the flaky test referenced by this issue should have been solved by and for non CSI /close
Closing this issue. <detailsIn response to : Im tracking these upgrade test failures elsewhere: They are not related to this original issue, the flaky test referenced by this issue should have been solved by /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
as FYI to call this out clearly in the Release notes.
Code change:
"fmt" "io" "os" <ins> "strconv" </ins> "github.com/GoogleCloudPlatform/kubernetes/pkg/api" cmdutil "github.com/GoogleCloudPlatform/kubernetes/pkg/kubectl/cmd/util"

Related discussion:
text: I an ELB to the Kibana cluster log viewer but it is not reported by which, aside from having a URL, should check each cluster service and if it has an external IP it should report that instead of the master proxy address.
cc
Code change:
} ) <ins> func TestConflictingCurrentContext(t *testing.T) { commandLineFile, _ := ioutil.TempFile("", "") defer os.Remove(commandLineFile.Name()) envVarFile, _ := ioutil.TempFile("", "") defer os.Remove(envVarFile.Name()) mockCommandLineConfig := clientcmdapi.Config{ CurrentContext: "any-context-value", } mockEnvVarConfig := clientcmdapi.Config{ CurrentContext: "a-different-context", } WriteToFile(mockCommandLineConfig, commandLineFile.Name()) WriteToFile(mockEnvVarConfig, envVarFile.Name()) loadingRules := ClientConfigLoadingRules{ CommandLinePath: commandLineFile.Name(), EnvVarPath:      envVarFile.Name(), } mergedConfig, err := loadingRules.Load() if err != nil { t.Errorf("Unexpected error: %v", err) } if mergedConfig.CurrentContext != mockCommandLineConfig.CurrentContext { t.Errorf("expected %v, got %v", mockCommandLineConfig.CurrentContext, mergedConfig.CurrentContext) } } </ins> func TestResolveRelativePaths(t *testing.T) { pathResolutionConfig1 := clientcmdapi.Config{ AuthInfos: map[string]clientcmdapi.AuthInfo{

Related discussion:
text: I've been pulling my hair out trying get kubectl talk to my local cluster. It seems that kubectl merges the local file with values pulled down from the cloud. I've done the following: The I just set is silently overridden. Questions: is this documented anywhere? should kubectl merge in my GCE cluster when I set my cloud-provider to something else? should kubectl override my previously set current-context?
workaround is to use flag to override the config file value.
but not a good UX long term
Someone else hit this as well: . Better fix is to have the local-up script set a context, which I will try to add today. As for the overwriting behavior, it's documented in , but based on what you describe, the intended load order is not working; local .kubeconfig should overwrite global. cc
That sounds infuriating. I'll take a look this morning.
Code change:
UpdateFake(obj *metav1.PartialObjectMetadata, opts metav1.UpdateOptions, subresources ...string) (*metav1.PartialObjectMetadata, error) } <ins> // NewTestScheme creates a unique Scheme for each test. func NewTestScheme() *runtime.Scheme { return runtime.NewScheme() } </ins> // NewSimpleMetadataClient creates a new client that will use the provided scheme and respond with the // provided objects when requests are made. It will track actions made to the client which can be checked // with GetActions().

Related discussion:
text: Because of this scheme mutation, this leads to concurrent use of the same scheme to cause the program to panic. It would be ideal to do this in some concurrent safe manner or add a warning or similar
/sig api-machinery
/assign
/assign /triage accepted
Code change:
cache.ResourceEventHandlerFuncs{UpdateFunc: vs.syncNodeZoneLabels}, zoneLabelsResyncPeriod, ) <ins> nodeLister := informerFactory.Core().V1().Nodes().Lister() vs.nodeManager.SetNodeLister(nodeLister) </ins> klog.V(4).Infof("Node informers in vSphere cloud provider initialized") }

Related discussion:
text: I am enabling CSI migration on a vSphere cluster. A node was fully drained, all its Pods are deleted, but just before KCM detaches in-tree vSphere volume volumes from it, KCM restarts. A new KCM starts, sees attached volumes in that are not needed by any Pod on the node and calls vSphere in-tree volume plugin . The in-tree vSphere volume plugin + cloud provider marks the volume as detached right away, without reaching to vCenter API. The volume remains attached forever. I don't think it's related to CSI migration, IMO a regular node drain + KCM restart at the same time can cause the same issue. KCM logs: The in-tree cloud provider has a and from it. It apparently has not processed all nodes at the time A/D controller tries to detach the node. The in-tree vSphere volume plugin + cloud provider detaches the volume in vCenter. That's really tricky, it's very hard / impossible to reproduce on a real cluster. I had to inject errors to Detach() in-tree volume call, so they fail until KCM is restarted. Newly started KCM would detach the volume just fine (no error would be injected there), but marks the volume as detached right away. For the record (and search-ability), the CSI driver that tries to attach the volume to another node complains either: Or: cc It could be somewhat related to , however, not all cases of can be related to KCM restart. As I mentioned above, it's very hard/impossible to synchronize KCM restart with draining a node in a real cluster. <details</details<detailsvSphere 7.0.3 </details<details</details<detailsOpenShift </details<detailsCRI-O </details<details</details
/sig storage
Tried just moving a Pod between nodes without CSI migration involved + restarting KCM when the volume is still attached to the old node. In-tree volume plugin on the new node says:
/assign
What CSI driver version and K8s version are you testing with?
Kubernetes is 1.25 and CSI driver is 2.7, still, I think it's not releated to the CSI driver, it's KCM who marks the in-tree volume as detached due to not populated node cache.
Code change:
cache.ResourceEventHandlerFuncs{UpdateFunc: vs.syncNodeZoneLabels}, zoneLabelsResyncPeriod, ) <ins> nodeLister := informerFactory.Core().V1().Nodes().Lister() vs.nodeManager.SetNodeLister(nodeLister) </ins> klog.V(4).Infof("Node informers in vSphere cloud provider initialized") }

Related discussion:
text: /assign
/triage accepted
I found an easier way how to reproduce this without extra code in KCM: a StatefulSet with 1 Pod that uses 1 PVC. the node where the Pod runs. down kube-controller-manager to zero (not really possible for me) or kill all kube-controller-manager Pods. This will cause new KCM processes to wait for leader election. In my env. this gives me ~30 seconds for the next step. Delete the Pod. A new one will be scheduled on a different node (because the current node is cordoned) + there is no KCM running at this time that would detach the volume from the old node. KCM wins its leader election, sees a volume attached to a node where it is not needed -it calls of the volume plugin. VSphere cloud provider does not have its caches populated yet (= this bug) -the volume is not detached, but returns success.
dangling attach does not help here. The volume is still attached to the old node even after ~10 minutes. fails with and not with . I can see there is a lot of dangling attach code in the cloud provider, but I did not debug further.
Code change:
// Status of containers in the pod. ContainerStatuses []*ContainerStatus // Status of the pod sandbox. <ins> // Only for kuberuntime now, other runtime may keep it nil. </ins> SandboxStatuses []*runtimeapi.PodSandboxStatus }

Related discussion:
text: What happened: It's most visible in the test <img width="323" alt="Screen Shot 2020-05-20 at 6 09 10 PM" src="https://user-"It shows up in kind, kube- and kops jobs The regression was introduced somewhere within , possible culprits are: (FYI (FYI (FYI There are more tests affected, just not as prominently (visually inspecting ) [sig-apps] StatefulSet [] Basic StatefulSet functionality [StatefulSetBasic] should perform rolling updates and roll backs of template modifications with PVCs [sig-apps] StatefulSet [] Basic StatefulSet functionality [StatefulSetBasic] should perform rolling updates and roll backs of template modifications [Conformance] [sig-storage] In-tree Volumes [Driver: nfs] [Testpattern: Inline-volume (default fs)] volumes should store data [sig-storage] In-tree Volumes [Driver: nfs] [Testpattern: Pre-provisioned PV (default fs)] volumes should store data I wasn't sure whether to tag this as kind/regression or not, I'll leave that up to or someone /sig node /sig api-machimery /sig scheduling
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : What happened: It's most visible in the test (FYI (FYI (FYI StatefulSet [] Basic StatefulSet functionality [StatefulSetBasic] should perform rolling updates and roll backs of template modifications with PVCs [sig-apps] StatefulSet [] Basic StatefulSet functionality [StatefulSetBasic] should perform rolling updates and roll backs of template modifications [Conformance] [sig-storage] In-tree Volumes [Driver: nfs] [Testpattern: Inline-volume (default fs)] volumes should store data [sig-storage] In-tree Volumes [Driver: nfs] [Testpattern: Pre-provisioned PV (default fs)] volumes should store data /sig api-machimery /sig scheduling Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.
Code change:
// Status of containers in the pod. ContainerStatuses []*ContainerStatus // Status of the pod sandbox. <ins> // Only for kuberuntime now, other runtime may keep it nil. </ins> SandboxStatuses []*runtimeapi.PodSandboxStatus }

Related discussion:
text: </details
/cc
will increase the time for deletion of pod and it should be expected because currently pod is deleted from apiserver before removing the sandbox container but after this fix pod will be deleted from the apiserver only after deleting the sandbox container.
/remove-sig scheduling doesn't affect scheduling running time at all.
Yeah, is almost definitely the culprit. Last time we changed that code path, it also caused a regression. The degradation in performance is expected, but a 5x slowdown is definitely larger than it should be.
do we know if that's because the status used by the path is stale, or if removing sandboxes is actually that slow?
if there's not a quick fix, should we revert and bring it back in once the root issue is understood?
I think a revert is appropriate in this case.
ok, I'll open it
open at
I agree with that 5x is something which shouldn't happen and i can take a look on why thats happening and that might require changing the way how the sandbox is currently deleted which is only during GC.
Code change:
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/util/wait" "k8s.io/client-go/kubernetes/fake" <del> "k8s.io/kubernetes/pkg/apis/apps" </del> kubepod "k8s.io/kubernetes/pkg/kubelet/pod" "k8s.io/kubernetes/pkg/kubelet/prober/results" "k8s.io/kubernetes/pkg/kubelet/status"

Related discussion:
text: The metric used to report acutal pod in the label , but now I see the podname prefix only. The full pod name in the label, so I can trace the metric to the failing pod. Check raw api and verify the actual pod name with uids. <details</detailsAzure AKS <details</detailsNo response No response No response_
/sig node
/triage accepted /priority important-soon /cc As discussed this is a change in functionaloty that wasn't highlighted in release notes. Need to decide on what do do with it. I understand it's a change of functionality. what is the side effect of this for you? Does the label without last part not informational enough?
I've been using the metric in my custom grafana dashboard that shows CPU, RAM, IO usage of pods with graph of increases in probes. If the apps are stable, the probe graph looks steadily. I can see if some irregularities in system metrics are influencing probes of respective pod. Also, if readiness probe is failing, the pod is removed from service, but is not restarted so I must have knowledge of pods with failing readiness probes. I attach the screenshot of my graph. ! So I'd welcome the possibility to pair the probes with exact pod and thank you for looking into it.
are we thinking that this resulted in an undesirable change? Spologies for being a bit late to reply, I'm returning from a brief paternity leave staycation.
This issue is labeled with but has not been updated in over 90 days, and should be re-triaged. Important-soon issues must be staffed and worked on either currently, or very soon, ideally in time for the next release. You can: Confirm that this issue is still relevant with (org members only) Deprioritize it with or Close this issue with For more details on the triage process, see /remove-triage accepted
Well this is awkward... I still believe the actual behavior does not respect the intended idea...
This issue makes it hard to link probe metrics with right pods. The value of pod label in and are also inconsistent with how it is used in other kubelet metrics, see: vs Definitely a bug, regression since 1.25.
I can't get anyone to look at the fix/revert: you may have better luck
/milestone v1.28
discussion in
You've pushed it through, congrats!
Code change:
expectedErrMsg:      fmt.Errorf("failure of getting instance metadata"), }, { <del> name:                "NodeAddresses should report error if VMSS instanceID is invalid", nodeName:            "vm123456", metadataName:        "vmss_$123", vmType:              vmTypeVMSS, useInstanceMetadata: true, expectedErrMsg:      fmt.Errorf("failed to parse VMSS instanceID %q: strconv.ParseInt: parsing %q: invalid syntax", "$123", "$123"), }, { </del> name:                "NodeAddresses should report error if cloud.vmSet is nil", nodeName:            "vm1", vmType:              vmTypeStandard,

Related discussion:
text: <!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: If computer name prefix does not match VMSS name, then Azure cloud provider will use VMSS API instead of using metadata service data. This leads to throttling, because every affected kubelet is making API requests. What you expected to happen: Azure cloud provider should use metadata service data instead of calling VMSS API when instance is current. How to reproduce it (as minimally and precisely as possible): Set computer name prefix of the VMSS to something different than default. You should observe Azure provider making requests to the VMSS API rather than using metadata service, which leads to throttling if you have multiple kubelets with this bug. Anything else we need to know?: This is bug was introduced by - returns false even if instance is current, because it expects node name to match VMSS name + instance id, however node name is derived from the computer-name-prefix which can be different than VMSS name. Node name should be matched against , not against . Environment: Kubernetes version (use ): 1.18.x and higher Cloud provider or hardware configuration: azure /sig cloud-provider
/sig cloud-provider
/area provider/azure
CC
/remove-sig cloud-provider /sig cloud-provider (trying to remove the label)
/sig cloud-provider
Thanks for the reporting. Unfortunately, computer name prefix is not included in IMDS, so it can't be used without VMSS API for constructing the NodeName. Would check what could we do to workaround the issue.
Can you memorize the computer name prefix in the first request to the Azure API and then use it for the isCurrentInstance check?
opened to fix the issue. Would you like to have a look?
I can confirm that your change should fix the problem I reported. Do you know if your fix will be to 1.18.x?
is working on the cherry pick
Thank you for quick fix!
Code change:
OOMScoreAdj:        config.OOMScoreAdj, ConfigSyncPeriod:   config.ConfigSyncPeriod.Duration, HealthzServer:      healthzServer, <ins> UseEndpointSlices:  false, </ins> }, nil }

Related discussion:
text: After this () has been merged and EnpointSlice feature is now enabled by default, we noticed Kube-proxy will not work on Windows. Only message I can see in the logs is: /sig windows
/cc /cc /cc
How about separating the above check into two if statements so that the log can pinpoint which field is false ?
It seems long term, EndpointSlice support should be so that is kept in sync with other implementations.
cc
Thanks for reporting this! Windows proxiers were still intended to be reading from Endpoints, but apparently that's not working as intended. EndpointSlices are moving back to disabled by default for 1.17, so that should fix the failing tests for now. I'll work on an actual fix for this as well, but it may not make it in until after KubeCon. /assign /sig network /priority important-soon
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry. ‍
/remove-triage unresolved
/assign cc
GitHub didn't allow me to assign the following users: jocelynberrendonner. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign cc Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
: Please let me know if you need any help on the fix! I'll be more than happy to review/help test the fix on Windows.
Code change:
}) It("should serve a basic image on each replica with a private image", func() { <del> switch testContext.Provider { case "gce", "gke": ServeImageOrFail(c, "private", "gcr.io/_b_k8s_authenticated_test/serve_hostname:1.1") default: </del> <ins> if !providerIs("gce", "gke") { </ins> By(fmt.Sprintf("Skipping private variant, which is only supported for providers gce and gke (not %s)", testContext.Provider)) <ins> return </ins> } <ins> ServeImageOrFail(c, "private", "gcr.io/_b_k8s_authenticated_test/serve_hostname:1.1") </ins> }) })

Related discussion:
text: I noticed while running e2e tests locally that some tests were apparently being incorrectly skipped: I checked, and this is also true for our Jenkins runs. Currently it looks like and have this bug, though only the former actually skips the test, I think. The issue appears to be that the code inside the top-level in these files is being run before the testContext struct is filled out (and before is even called), and so testContext.Provider is empty. I'm not immediately sure what the proper fix is; I'm also worried that there might be other things broken by the fact that these blocks are called before initialization is complete. We also probably investigate if there is a better way to mark these tests as skipped; right now, there's no indication in the test report (, Jenkins test report, etc) that these tests were skipped.
I merged and which tried to disable these tests for non-{gke,gce} but apparently didn't follow-up and verify that they were still running on our platforms. /cc
Seems easy enough to fix. I'll have a PR out to fix this before Monday.