jiebi/CodeConvo huggingface dataset:
Code change:
storageframework.BlockVolModeDynamicPV, storageframework.Ext4DynamicPV, storageframework.XfsDynamicPV, <ins> storageframework.NtfsDynamicPV, </ins> } return InitCustomMultiVolumeTestSuite(patterns) }

Related discussion:
text: After a restart, we would definitely not add a pod back that was in terminating state into desired state of the world, but "restart + this change" should be roughly equivalent to "restart and force delete while restart is processing" The other change was in which changed a live CRI read (which was trying to enforce that the dsowp is synchronized with the pod worker) into a threadsafe check against the pod worker directly (the pod worker can provide authoritative answers to "this pod has no running containers" now). The latter change seems less likely to impact you.
Thank you for the details! I think your change is making the right change, but unfortunately some of volume management still have issues, and so the tests rely on some behaviors that is not very right (pod volume is back to desired state even after it is in terminating) The cleanup mount currently does not have the logic to clean up global mount point if the volume plugin sets it up. I am still testing more and will share more details.
is this a release blocker?
Yes, this impacts volume cleanup if kubelet restarts for any reason.
On a kube-up cluster running the latest master against the latest e2e binary, I was able to reproduce the in-tree failures but not the CSI ones for some reason. Will focus on in-tree failures for the time being
/triage accepted
marking release-blocker per note on slack:
/priority critical-urgent (once upon a time we used to not ship a release if any priority/critical-urgent issues were open, not sure if we decided to stop doing that)
We were able to find the trigger for the CSI failures - on 6/29 there was a change in the hostpath driver test manifest, and the driver's is enabled. This makes the driver return an error instead of only logging a warning message on if the volume isn't unstaged properly. IIUC there's a long standing issue that reconstruction doesn't handle node unstage. could you confirm? If that's the case, then these CSI failures are not release blocking. Furthermore, the CSI test with graceful delete started to fail on 7/9 because of since graceful and force deletes are now considered the same from the point of view of the volume manager. cc
When that check was enabled, I don't think we realized it would impact the reconstruction path; it was enabled to validate a different issue.
Code change:
storageframework.BlockVolModeDynamicPV, storageframework.Ext4DynamicPV, storageframework.XfsDynamicPV, <ins> storageframework.NtfsDynamicPV, </ins> } return InitCustomMultiVolumeTestSuite(patterns) }

Related discussion:
text: Are we running into the issue that reconstruction of staging directories isn't supported () OR is it that our reconstruction path requires drivers to have real mount points ()? Either way since both are known gaps with reconstruction, I think it makes sense to to disable the flag until we can address those issues.
I was looking at this today too, trying to understand the underlying cause. That is why the failures are being reported now, but the part that I didn't understand is why NodeUnstageVolume is not being called for these volumes. In this instance: The DeleteVolume request fails because a staged path is still present: And in the kubelet log, NodeUnpublishVolume is called Then we hit this error: And NodeUnstageVolume is never called for this volume (same for other failures I looked at) so DeleteVolume keeps failing. So I think you're right that it's triggered by reconstruction, and it looks similar to , except in this case the restart happened before NodeUnpublishVolume.
And it's probably not since the test uses a PVC
Will look deeper into why UnmountDevice wasn't issued as well, but some update on the in-tree tests: the force delete test is skipped for the in-tree hostpath CSI driver because of Graceful tests are now failing as well because they behave like force tests due to
I don't think the issue is related to PVC or not. I think it's related to drivers that do not actually mount devices as a separate filesystem than the root.
There are three possible flows during kubelet restart pod gracefully delete case, the pod still exists (it is not considered as terminated in function isPodTerminated(); note that this logic has some changes after , see , so pod volume will be back to desired state, and reconciler will try to set up volume and fill in actual state. When pod volume is finally deleted from desired state, reconciler will trigger teardown with information from actual state to unpublish and unstage volume. pod is forcefully deleted, pod volume will not be in desired state. If volume reconstruction succeeds (be able to reconstruct volume spec), volume will be back into actual state. Then reconciler will trigger teardown with information from actual state to unpublish and unstage volume (this part is similar to case 1).
Code change:
storageframework.BlockVolModeDynamicPV, storageframework.Ext4DynamicPV, storageframework.XfsDynamicPV, <ins> storageframework.NtfsDynamicPV, </ins> } return InitCustomMultiVolumeTestSuite(patterns) }

Related discussion:
text: last case is reconstruction failed (in this csi-host-path test, reconstruction will fail due to issue related to , it fail to check mount point for csi hostpath driver because it is using a bind mount of the same device ). In this case, reconciler will try to clean up mounts directly, but only published volume path, not the staged volume path (this logic is not implemented due to missing information). As a workaround, we can now disable check-volume-lifecycle if it does not cause other issues. Then we will work on to make sure csi-host-path driver can pass volume mountpoint check during reconstruction.
This issue is now mitigated with merged. There are two longer-term issues that are isolated to hostpath plugins (and maybe other plugins that are similar in behavior, perhaps emptydir and similar?): related to IsLikelyNotMountPoint as mentioned. Because these don't impact the core volume machinery and are not regressions, this issue doesn't need to be release blocking. We'll close this issue once (2) is tracked.
is the IsLikelyNotMountPoint issue
after is merged, two csi-hostpath driver subpath test can pass, and two in-tree driver gracefully delete tests were skipped. https://k8s-
can you please cherry-pick to release-1.22 branch?
Opened cherry-pick, please review:
- what is the plan for reenabling them?
cherry-pick merged!
Cheng and I will work on solving IsLikelyNotMountPoint issue for hostpath drivers. That should allow test to pass and we can enable them.
Can this be removed from the milestone now that the workarounds have been merged, while leaving this issue open to track further work?
We can use existing issue to track the real fix. /close
Closing this issue. <detailsIn response to : We can use existing issue to track the real fix. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Code change:
for k, tc := range testcases { tcName := k path := "/apis/" + tc.APIService.Spec.Group + "/" + tc.APIService.Spec.Version + "/foo" <del> called := false </del> <ins> timesCalled := int32(0) </ins> func() { // Cleanup after each test case. backendHandler := http.NewServeMux() backendHandler.Handle(path, websocket.Handler(func(ws *websocket.Conn) { <ins> atomic.AddInt32(&timesCalled, 1) </ins> defer ws.Close() body := make([]byte, 5) ws.Read(body) ws.Write([]byte("hello " + string(body))) <del> called = true </del> })) backendServer := httptest.NewUnstartedServer(backendHandler)

Related discussion:
text: There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig api-machinery
Code change:
obj.MakeIPTablesUtilChains = boolVar(true) } if obj.IPTablesMasqueradeBit == nil { <del> temp := int32(defaultIPTablesMasqueradeBit) </del> <ins> temp := int32(DefaultIPTablesMasqueradeBit) </ins> obj.IPTablesMasqueradeBit = &temp } if obj.IPTablesDropBit == nil { <del> temp := int32(defaultIPTablesDropBit) </del> <ins> temp := int32(DefaultIPTablesDropBit) </ins> obj.IPTablesDropBit = &temp } if obj.CgroupsPerQOS == nil {

Related discussion:
text: Solution: I will add one.
/cc
xref
Thanks for sending a PR so quickly!
[MILESTONENOTIFIER] Milestone Issue Needs Attention Action required: During code slush, issues in the milestone should be in progress. If this issue is not being actively worked on, please remove it from the milestone. If it is being worked on, please add the label so it can be tracked with other in-flight issues. Note: If this issue is not resolved or labeled as by Wed, Nov 22 it will be moved out of the v1.9 milestone. <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
Code change:
// This ensures the given VM's Primary NIC's Primary IP Configuration is // participating in the specified LoadBalancer Backend Pool. func (az *Cloud) ensureHostInPool(serviceName string, nodeName types.NodeName, backendPoolID string) error { <ins> var machine compute.VirtualMachine </ins> vmName := mapNodeNameToVMName(nodeName) az.operationPollRateLimiter.Accept() machine, err := az.VirtualMachinesClient.Get(az.ResourceGroup, vmName, "") if err != nil { <del> return err </del> <ins> if az.CloudProviderBackoff { glog.V(2).Infof("ensureHostInPool(%s, %s, %s) backing off", serviceName, nodeName, backendPoolID) machine, err = az.VirtualMachineClientGetWithRetry(az.ResourceGroup, vmName, "") if err != nil { glog.V(2).Infof("ensureHostInPool(%s, %s, %s) abort backoff", serviceName, nodeName, backendPoolID) return err } } else { return err } </ins> } primaryNicID, err := getPrimaryInterfaceID(machine)

Related discussion:
text: <!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: In certain large cluster scenarios we would like to anticipate kubelet behavior hitting request limits, and backoff in response. E.g., What you expected to happen: This is a request for add'l backoff coverage, so nothing surprising. How to reproduce it (as minimally and precisely as possible): Build a cluster with as many nodes as your subscription will support on azure until kubelet behavior contributes to backend API request limits. Environment: Kubernetes version (use ): and Cloud provider or hardware configuration**: azure OS (e.g. from /etc/os-release): Ubuntu 16.04.2 LTS (Xenial Xerus) Kernel (e.g. ): Linux k8s-pool7--7 4.4.0-83-generic -Ubuntu SMP Mon Jun 26 17:54:43 UTC 2017 x8664 x8664 x86_64 GNU/Linux Install tools: acs-engine Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig azure
Code change:
// GetPrimaryInterface gets machine primary network interface by node name and vmSet. func (ss *scaleSet) GetPrimaryInterface(nodeName, vmSetName string) (network.Interface, error) { <del> ssName, instanceID, vm, err := ss.getVmssVM(nodeName) </del> <ins> managedByAS, err := ss.isNodeManagedByAvailabilitySet(nodeName) </ins> if err != nil { <del> if err == ErrorNotVmssInstance { glog.V(4).Infof("GetPrimaryInterface: node %q is managed by availability set", nodeName) // Retry with standard type because nodes are not managed by vmss. return ss.availabilitySet.GetPrimaryInterface(nodeName, "") } </del> <ins> glog.Errorf("Failed to check isNodeManagedByAvailabilitySet: %v", err) return network.Interface{}, err } if managedByAS { // vm is managed by availability set. return ss.availabilitySet.GetPrimaryInterface(nodeName, "") } </ins> <del> glog.Errorf("error: ss.GetPrimaryInterface(%s), ss.getCachedVirtualMachine(%s), err=%v", nodeName, nodeName, err) </del> <ins> ssName, instanceID, vm, err := ss.getVmssVM(nodeName) if err != nil { glog.Errorf("error: ss.GetPrimaryInterface(%s), ss.getVmssVM(%s), err=%v", nodeName, nodeName, err) </ins> return network.Interface{}, err }

Related discussion:
text: <!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: VMAS nodes are reporting error in vmss cluster (vmType set to vmss). What you expected to happen: Node should register correctly in the kubernetes cluster. How to reproduce it (as minimally and precisely as possible): Create a VMAS nodes inside vmss cluster. Anything else we need to know?: Environment: Kubernetes version (use ): master Cloud provider or hardware configuration: azure OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig azure
Code change:
AppArmor:                                    {Default: true, PreRelease: utilfeature.Beta}, DynamicKubeletConfig:                        {Default: false, PreRelease: utilfeature.Alpha}, KubeletConfigFile:                           {Default: false, PreRelease: utilfeature.Alpha}, <del> DynamicVolumeProvisioning:                   {Default: true, PreRelease: utilfeature.Alpha}, </del> ExperimentalHostUserNamespaceDefaultingGate: {Default: false, PreRelease: utilfeature.Beta}, ExperimentalCriticalPodAnnotation:           {Default: false, PreRelease: utilfeature.Alpha}, Accelerators:                                {Default: false, PreRelease: utilfeature.Alpha},

Related discussion:
text: Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: kube-apiserver --help still shows this: It looks like the related feature can be turned off. However, dynamic volume provisioning Alpha has retried in 1.6. What you expected to happen: Remove from feature gate to avoid confusion. How to reproduce it (as minimally and precisely as possible): Run Anything else we need to know?: No Environment: Kubernetes version (use ): 1.7+ Cloud provider or hardware configuration**: Any OS (e.g. from /etc/os-release): Any Kernel (e.g. ): Any Install tools: Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig api-machinery
/sig api-machinery
SIG API Machinery doesn't own the contents of the API. /sig storage
Code change:
tags = ["automanaged"], visibility = ["//visibility:public"], ) <ins> go_test( name = "go_default_test", srcs = ["testutil_test.go"], embed = [":go_default_library"], deps = ["//staging/src/k8s.io/component-base/metrics:go_default_library"], ) </ins>

Related discussion:
text: What would you like to be : Once a metric be deprecated, we should update the unit test accordingly if there is any. We need to work out a way for a unit test freely select codebase version, and also should make sure the unit test won't fail in the subsequent release. Why is this needed: Deprecated also needs to be protected. /sig instrumentation /assign
Code change:
return GatherAndCompare(registry, expected, metricNames...) } <ins> // NewFakeKubeRegistry creates a fake `KubeRegistry` that takes the input version as `build in version`. // It should only be used in testing scenario especially for the deprecated metrics. // The input version format should be `major.minor.patch`, e.g. '1.18.0'. func NewFakeKubeRegistry(ver string) metrics.KubeRegistry { backup := metrics.BuildVersion defer func() { metrics.BuildVersion = backup }() metrics.BuildVersion = func() apimachineryversion.Info { return apimachineryversion.Info{ GitVersion: fmt.Sprintf("v%s-alpha+1.12345", ver), } } return metrics.NewKubeRegistry() } </ins>

Related discussion:
text: What would you like to be : Once a metric be deprecated, we should update the unit test accordingly if there is any. We need to work out a way for a unit test freely select codebase version, and also should make sure the unit test won't fail in the subsequent release. Why is this needed: Deprecated also needs to be protected. /sig instrumentation /assign
Code change:
containers, _ := selectContainers(spec.Containers, o.ContainerSelector) if len(containers) != 0 { for i := range containers { <del> if len(o.Limits) != 0 && len(containers[i].Resources.Limits) == 0 { containers[i].Resources.Limits = make(api.ResourceList) } for key, value := range o.ResourceRequirements.Limits { containers[i].Resources.Limits[key] = value } if len(o.Requests) != 0 && len(containers[i].Resources.Requests) == 0 { containers[i].Resources.Requests = make(api.ResourceList) } for key, value := range o.ResourceRequirements.Requests { containers[i].Resources.Requests[key] = value } </del> <ins> containers[i].Resources = o.ResourceRequirements </ins> transformed = true } } else {

Related discussion:
text: from https://k8s-
Really suspect this is
This is putting us into test-go post-commit run is broken
cc:
marked this p0 - if a fix isn't easy let's try to identify what introduced this and revert.
This flake on: 2016-10-30 07:19 PDT First instance: : It is flaking frequently, blocking PRs from getting merged, and slowing the submit queue ( at 75%). As on-call, I'm going to try and identify the cluprit and revert. Can use all help in doing so.
can you investigate whether your additional tests could cause this (by breaking the deployment of those pods, for instance) , Saad Ali wrote:
PR looks most suspect to me. It mucked with , and the errors we are seeing in each of the failures is related to : That PR was merged at Oct 30, 2016, 6:18 AM PDT and the first flake in was at Oct 30, 2016, 7:19 AM PDT. Thoughts? CC
I'm going to initiate a revert of to see if that fixes the issue
IMO we can just change the flaky test to not get pods but some other resources, such as deployments. We can't really expect to clean up pods immediately.
I'm going to proceed with revert (). Please feel free to send a follow up PR to handle this more gracefully.
Going to leave this opened until we have a few hours of runs with the revert. Want to verify, with some degree of certainty, that the revert took care of this flake.
We had 15 green runs of in a row. Considering this flake fixed.
Code change:
# DO NOT REPORT SECURITY VULNERABILITIES DIRECTLY TO THESE NAMES, FOLLOW THE # INSTRUCTIONS AT https://kubernetes.io/security/ <del> cjcullen joelsmith liggitt philips </del> <ins> cheftako deads2k lavalamp </ins> sttts <del> tallclair </del>

Related discussion:
text: The purpose of the file for each Kubernetes repository is to provide a list of people who can assist the Kubernetes in the event that a security issue related to the repository is discovered or disclosed. As described in the file, those on the list should agree to our . Please update the  file for the sample-controller repository. After finding people who are willing to work in this capacity, you should add them to the list, then remove PSC members (except any PSC member will be working as a security contact for this repository). The list is GitHub usernames, optionally followed by an email address. If no email address is listed, the PSC will use the email address found on git commits made by the listed user. The file may already have people listed who are secuirty contacts. In that case, simply remove any PSC members who aren't also security contacts for the repo. See for more information /area security /committee product-security /kind cleanup /lifecycle frozen /priority important-soon /sig api-machinery
Code change:
# DO NOT REPORT SECURITY VULNERABILITIES DIRECTLY TO THESE NAMES, FOLLOW THE # INSTRUCTIONS AT https://kubernetes.io/security/ <del> cjcullen joelsmith liggitt philips </del> <ins> cheftako deads2k lavalamp </ins> sttts <del> tallclair </del>

Related discussion:
text: The purpose of the file for each Kubernetes repository is to provide a list of people who can assist the Kubernetes in the event that a security issue related to the repository is discovered or disclosed. As described in the file, those on the list should agree to our . Please update the  file for the sample-apiserver repository. After finding people who are willing to work in this capacity, you should add them to the list, then remove PSC members (except any PSC member will be working as a security contact for this repository). The list is GitHub usernames, optionally followed by an email address. If no email address is listed, the PSC will use the email address found on git commits made by the listed user. The file may already have people listed who are secuirty contacts. In that case, simply remove any PSC members who aren't also security contacts for the repo. See for more information /area security /committee product-security /kind cleanup /lifecycle frozen /priority important-soon /sig api-machinery
Code change:
# DO NOT REPORT SECURITY VULNERABILITIES DIRECTLY TO THESE NAMES, FOLLOW THE # INSTRUCTIONS AT https://kubernetes.io/security/ <del> cjcullen joelsmith liggitt philips </del> <ins> cheftako deads2k lavalamp </ins> sttts <del> tallclair </del>

Related discussion:
text: The purpose of the file for each Kubernetes repository is to provide a list of people who can assist the Kubernetes in the event that a security issue related to the repository is discovered or disclosed. As described in the file, those on the list should agree to our . Please update the  file for the kube-aggregator repository. After finding people who are willing to work in this capacity, you should add them to the list, then remove PSC members (except any PSC member will be working as a security contact for this repository). The list is GitHub usernames, optionally followed by an email address. If no email address is listed, the PSC will use the email address found on git commits made by the listed user. The file may already have people listed who are secuirty contacts. In that case, simply remove any PSC members who aren't also security contacts for the repo. See for more information /area security /committee product-security /kind cleanup /lifecycle frozen /priority important-soon /sig api-machinery