jiebi/CodeConvo huggingface dataset:
Code change:
for _, test := range storagePodsTests { t.Run(test.name, func(t *testing.T) { <ins> for k, v := range test.features { defer featuregatetesting.SetFeatureGateDuringTest(t, feature.DefaultFeatureGate, k, v)() } </ins> node := v1.Node{Status: v1.NodeStatus{Capacity: makeResources(10, 20, 32, 5, 20, 5).Capacity, Allocatable: makeAllocatableResources(10, 20, 32, 5, 20, 5)}} test.nodeInfo.SetNode(&node)

Related discussion:
text: What happened: In a v1.18 cluster with the LocalStorageCapacityIsolation feature gate disabled setting an ephemeral-storage request or limit causes the pod to be unschedulable: If the ephemeral-storage request is removed the pod is scheduled. This behavior exists in v1.19 and a recent v1.20 build. What you expected to happen: In previous releases such pods are scheduled. I expect that any ephemeral-storage limit is not enforced. How to reproduce it (as minimally and precisely as possible): Disable the LocalStorageCapacityIsolation feature guide by adding to the kube-scheduler. Create a pod like this: Anything else we need to know?: The problem was originally seen after moving from IBM Cloud Red Hat OpenShift v4.4 to v4.5. Pods that worked in 4.4 (Kubernetes v1.17) remained in a Pending state in OpenShift v4.5 (Kubernetes 1.18). We discovered that the IBM Cloud OpenShift service has the LocalStorageCapacityIsolation feature-gate disabled while Red Hat has enabled the feature. We then replicated the behavior with a plain Kubernetes cluster. Environment: Kubernetes version (use ): v1.18.10, v1.19.3, v1.20.0 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig sig-scheduling
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig sig-scheduling Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling
I can confirm it's a bug. Will work on it.
Code change:
for _, test := range storagePodsTests { t.Run(test.name, func(t *testing.T) { <ins> for k, v := range test.features { defer featuregatetesting.SetFeatureGateDuringTest(t, feature.DefaultFeatureGate, k, v)() } </ins> node := v1.Node{Status: v1.NodeStatus{Capacity: makeResources(10, 20, 32, 5, 20, 5).Capacity, Allocatable: makeAllocatableResources(10, 20, 32, 5, 20, 5)}} test.nodeInfo.SetNode(&node)

Related discussion:
text: /assign
Clarifying the recreate steps: I disabled the LocalStorageCapacityIsolation feature guide by adding to the kube-scheduler since this behavior seems to be within the scheduler.
Thanks for confirming!
This bug has been fixed in master branch, and will be backported to [1.18] and [1.19] soon. [1.18]: [1.19]:
Code change:
if filePath == m.checkpointFile() { continue } <del> stat, err := os.Stat(filePath) </del> <ins> // TODO: Until the bug - https://github.com/golang/go/issues/33357 is fixed, os.stat wouldn't return the // right mode(socket) on windows. Hence deleting the file, without checking whether // its a socket, on windows. stat, err := os.Lstat(filePath) </ins> if err != nil { klog.Errorf("Failed to stat file %s: %v", filePath, err) continue

Related discussion:
text: <!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: Flakes: Full failure: Which test(s) are failing: Since when has it been failing: Since merging Testgrid link: Reason for failure: Serial slow tests are failing with: and the parallel tests are flaking with: Anything else we need to know: I would not expect kubelet to crash if the node if the plugin is not present. /sig windows cc:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
It seems to me that it is not related to a plugin being not present, but that it cannot open the endpoint where potential plugins could register to. In other words, I suspect this error happens independently from any plugins. If we can trust this message , then this would indicate that it tries to open that listening socket twice. This would mean that is called twice. If this is the case, the question would be: Why? It would be interesting to see if shows twice.
There is an issue on some Windows not updated. When you restart kubelet and the socket file So the workarround, is to remove before restart kubeblet
And why tests on the PR didn't trigger this error ?
Question: Is it legal that is called twice? It seems that the function is called from , which is analogous to the Linux implementation Calling twice looks fishy to me. It seems to me that this problem is not caused by, but only exposed by the device plugin change
You mean that kubelet is initialized twice on the same computer, causing this error the second time? How is this handled on Linux? If this is allowed we could simply delete if present before opening it again.
Are you sure it's call twice ? because on windows is not present but replace by you can see the first line comment and .
Code change:
if filePath == m.checkpointFile() { continue } <del> stat, err := os.Stat(filePath) </del> <ins> // TODO: Until the bug - https://github.com/golang/go/issues/33357 is fixed, os.stat wouldn't return the // right mode(socket) on windows. Hence deleting the file, without checking whether // its a socket, on windows. stat, err := os.Lstat(filePath) </ins> if err != nil { klog.Errorf("Failed to stat file %s: %v", filePath, err) continue

Related discussion:
text: A simple test is: kubelet and check the socket file is still present kubelet you got the error, stop it socket file kubelet again, normally it's working Maybe the fix can be add in
I think your explanation is very plausible: That it is not called twice within a single kubelet initialization, but kubelet is initialized again for some reason on the same node. (It still means that it is called twice, but I didn't think about the possibility that kubelet is initialized twice.)
There are various ways of fixing it. But I would be interested in learning how this is handled under Linux. To choose the same path for Windows, too.
Thanks for your quick response and analysis :+1: It would be interesting to see if shows twice. You can find the kubelet logs (and many others) for the windows nodes in the "artifacts" tab on This set of tests runs configuration for GMSA tests which as part of the set up of the domain controller, causes a node resets. This explains why one of the nodes seems ok (it isn't randomly selected as part of the domain machines) and why the parallel tests don't cause a full failure (they don't cause node resets). Not all clusters use that script. This set of tests don't use them either (uses aks-engine which has a different start up script). IMHO this is a breaking change scince any cluster that has a node restart would fail to come back online. and kubelet should be able to restart (because of node restart or crash for another reason, etc.) It looks like there is code to clean up old sockets but it is not working on windows: Given there is clean up code there and it's a breaking change I think this this should be fixed in kubelet.
Oh yes, I missed that! (I should have looked only a few lines above the line causing the error.) It at least looks exactly like the code we need. The function is implemented in the same file: But why it is failing to work under Windows? Do you have an idea? Yes, sounds good.
Ok so on windows failed here
So this exemple reproduce the error. the failed only on socket file not on regular file ! Link to
It's bug on windows 1903 not present on windows 1809 The fix can be replace by
Very cool!
Code change:
if filePath == m.checkpointFile() { continue } <del> stat, err := os.Stat(filePath) </del> <ins> // TODO: Until the bug - https://github.com/golang/go/issues/33357 is fixed, os.stat wouldn't return the // right mode(socket) on windows. Hence deleting the file, without checking whether // its a socket, on windows. stat, err := os.Lstat(filePath) </ins> if err != nil { klog.Errorf("Failed to stat file %s: %v", filePath, err) continue

Related discussion:
text: Just for interest, how did you find
I hate Windows ;-) and I googled
It's bug on windows 1903 golang/go (comment) not present on windows 1809 This set of tests are running on 2019 LTSC (also known as 1809). If this is only an issue on 1903 is there something else happening?
Sorry, I read to quickly, there is a change of behaviour between 1809 and 1903 about unix socket, If my understanding is good on 1903 unix socket is like regular file but not on 1809, so it's why os.Stat failed on 1809. but I tested on Windows 10 20H2 and the issue still here with
So in summary, the key point is: PR fixes this issue
Code change:
github.com/satori/go.uuid v1.2.0 // indirect github.com/stretchr/objx v0.1.1 // indirect github.com/stretchr/testify v1.2.2 <del> github.com/vmware/govmomi v0.20.0 </del> <ins> github.com/vmware/govmomi v0.20.1 </ins> golang.org/x/crypto v0.0.0-20181025213731-e84da0312774 golang.org/x/oauth2 v0.0.0-20190402181905-9f3314589c9a google.golang.org/api v0.0.0-20181220000619-583d854617af

Related discussion:
text: <!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: The vSphere cloud provider is unable to fetch the node failure domain labels when using VCP with SAML auth because of an authentication failure. For context, the SAML token auth is described here - SAML auth support for the zones api was introduced in this PR - , which was cherry-picked to 1.14.1 here - I've tested the credentials I'm using with govc to ensure that I'm able to get a token: The credentials also work fine when using VCP without the node labels feature. Relevant kubelet logs: What you expected to happen: VCP should add failure domain labels to the node object. How to reproduce it (as minimally and precisely as possible): Setup SAML auth for VCP with zone labeling enabled. Anything else we need to know?: There was a previous issue that was raised () but that was before SAML auth support for zones was implemented. This could potentially be an issue with govc handling of SAML tokens since even with the Administrator role assigned to the user, I'm still unable to list tags. The role and keys were generated with the following: Environment: Kubernetes version (use ): 14.1 Cloud provider or hardware configuration: vSphere OS (e.g: ): Linux & Windows
/sig vmware
Would you mind taking a look at this? I can provide more info if needed.
govc's command only works with the SOAP endpoint currently. The REST endpoint has its own session manager and requires a separate login. At the moment, govc's tags commands login/logout when each command is run and don't use a persisted session (which are saved to ~/.govmomi/sessions). However, you can use a token returned by to test SAML auth with the tags commands. For example: Can you give that try? There are some TODOs around this btw:
Those commands work for me. So that would imply the solution user, role, and permissions are set up correctly? Would you expect the to also work?
Code change:
github.com/satori/go.uuid v1.2.0 // indirect github.com/stretchr/objx v0.1.1 // indirect github.com/stretchr/testify v1.2.2 <del> github.com/vmware/govmomi v0.20.0 </del> <ins> github.com/vmware/govmomi v0.20.1 </ins> golang.org/x/crypto v0.0.0-20181025213731-e84da0312774 golang.org/x/oauth2 v0.0.0-20190402181905-9f3314589c9a google.golang.org/api v0.0.0-20181220000619-583d854617af

Related discussion:
text: I seem to be getting a 404 when using with the flag - Does seem to work when specifying a tag name: Are there any other govc commands I should check / verify?
yes, the solution user setup looks right. Not sure what the issue is with . Can you share your vCenter build number ()? I'll try to reproduce the issue.
$ govc about -u vcdo Name: VMware vCenter Server Vendor: VMware, Inc. Version: 6.7.0 Build: OS type: linux-x64 API type: VirtualCenter API version: 6.7.2 Product ID: vpx UUID: -55ed-4737-9cc4-
I can reproduce this, but don't know what the issue is yet. The version of Go and vSphere have changed since , but if it was one of those was the cause, I'd expect SAML auth to fail with the govc tags commands too. All I have to go on for now is the vCenter log message below. I'll update when I have more info.
I have it working with the change below, but need to figure out a few things first: Why this is the case when run inside kubelet and not govc (note that the port change to 443 breaks govc) A good way to determine which port to use (depends on the answer to the above) If this is behavior is specific to 6.7U2 or something triggered by a change in K8s I suppose a quick fix would change to try port 443 if 80 fails, but would prefer to avoid that as the failure generates stack traces in the
/assign
This is obvious in retrospect, but the issue is the message signature needs to use the same port as the original request URL. However, in the govc case there was no port set in the request URL. The client defaults to port when connecting (based on the scheme), but does not include any port with the header. In the case, the default is on the server side. This can be reproduced with if includes the port , rather than just as we were using. The VCP tests that use token auth for tags lookup just don't validate the signature, so this wasn't a change related to 6.7U2.
Code change:
github.com/satori/go.uuid v1.2.0 // indirect github.com/stretchr/objx v0.1.1 // indirect github.com/stretchr/testify v1.2.2 <del> github.com/vmware/govmomi v0.20.0 </del> <ins> github.com/vmware/govmomi v0.20.1 </ins> golang.org/x/crypto v0.0.0-20181025213731-e84da0312774 golang.org/x/oauth2 v0.0.0-20190402181905-9f3314589c9a google.golang.org/api v0.0.0-20181220000619-583d854617af

Related discussion:
text: I'll open a K8s PR to pull in the fix:
Updating k/vendor to govmomi master adds almost 6K lines for a 3 line fix. I may just create a govmomi branch off of v0.20 to reduce churn in the K8s release branches for this fix, may have to wait until tomorrow.
Really appreciate the insight into this issue. Can confirm that specifying the port (443) in the GOVC_URL reproduces the issue. Thank you!
Unfortunately, I'm seeing the same issue with v1.14.3: cloud_config Works fine when I use the username and password of a vsphere admin, but using cert and key still has authentication issues.
any related errors in vCenter's ?
Seeing a lot of this exception:
whoa! that stack trace is like punch in the gut (brings back memories!) :)
indeed!! apologies, I must have had the change for PR locally, but failed to include it as I had focused on the signature issue and updating vendor/
Code change:
location.Scheme = "https" rloc, err := r.serviceResolver.ResolveEndpoint(handlingInfo.serviceNamespace, handlingInfo.serviceName) if err != nil { <del> http.Error(w, fmt.Sprintf("missing route (%s)", err.Error()), http.StatusInternalServerError) </del> <ins> glog.Errorf("error resolving %s/%s: %v", handlingInfo.serviceNamespace, handlingInfo.serviceName, err) http.Error(w, "service unavailable", http.StatusServiceUnavailable) </ins> return } location.Host = rloc.Host

Related discussion:
text: https://k8s- https://k8s- https://k8s- https://k8s- https://k8s- ... A lot of them /cc
/assign
Still see this failure after is merged, although less frequently: https://k8s- https://k8s- Reopen.
Looks like the client doesn't differentiate between 500 and 503 here: It should. Will open a fix shortly
opened
fix merged, will monitor https://k8s-
Thanks! :)
haven't seen this in the 12 hours since the fix merged, was seeing it every 2-3 hours before. /close
Code change:
t.Errorf("Expect an NotFound error, got %v", err) } } <ins> func TestPrefix(t *testing.T) { scheme, codecs := testScheme(t) codec := apitesting.TestCodec(codecs, examplev1.SchemeGroupVersion) server := etcdtesting.NewEtcdTestClientServer(t) defer server.Terminate(t) testcases := map[string]string{ "custom/prefix":     "/custom/prefix", "/custom//prefix//": "/custom/prefix", "/registry":         "/registry", } for configuredPrefix, effectivePrefix := range testcases { helper := newEtcdHelper(server.Client, scheme, codec, configuredPrefix) if helper.pathPrefix != effectivePrefix { t.Errorf("configured prefix of %s, expected effective prefix of %s, got %s", configuredPrefix, effectivePrefix, helper.pathPrefix) } } } </ins>

Related discussion:
text: an API server with , populate with data, stop server etcd2 to etcd3 the API server with data is missing from the API. The etcd2 client takes the configured prefix and passes it to the etcd client as . The etcd3 client simply passes in the configured prefix. is a no-op on the default prefix (`/registry), but can modify custom prefixes that did not start with a slash. cc
I think we have to preserve the behavior from the etcd2 client setup. It's a no-op for the default prefix, but requiring a change to a custom prefix when switching from etcd2 to etcd3 is really unexpected, especially when the failure mode is that the apiserver starts up as if it's a brand new cluster (including creating default projects, kubernetes service, etc)
cc
/cc re: upgrade testing.
Throwing release blocker on here just so we don't lose it (given data loss potential)
- thank for debugging and fixing it.
Code change:
masqueradeValue := 1 << uint(masqueradeBit) masqueradeMark := fmt.Sprintf("%#08x/%#08x", masqueradeValue, masqueradeValue) <del> if nodeIP == nil { klog.Warningf("invalid nodeIP, initializing kube-proxy with 127.0.0.1 as nodeIP") nodeIP = net.ParseIP("127.0.0.1") } </del> isIPv6 := utilnet.IsIPv6(nodeIP) klog.V(2).Infof("nodeIP: %v, isIPv6: %v", nodeIP, isIPv6)

Related discussion:
text: <!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Using Kubernetes 1.16.1, I started kube-proxy in iptables mode as a static pod on a master node. The master node was intentionally (and permanently) not available in the API as a Node object. kube-proxy started up and terminated soon after because it failed to (which is supposed to function as the bound address) and eventually returned an error: What you expected to happen: I expected the same behavior as in Kubernetes 1.15 where failure to look up the Node in the API would result in . How to reproduce it (as minimally and precisely as possible): Configure kube-proxy with iptables mode on Kubernetes 1.16 without the hosting Node being available in the Kubernetes API (under the OS hostname). Anything else we need to know?: PR (Fix for kube-proxy to wait for some duration for the node to be defined) introduced an additional . I believe this is the part that is causing the change in behavior: in Kubernetes 1.15, the check would be omitted and cause to set the default bind address to 127.0.0.1 (when iptables mode is set; I haven't checked other modes). The PR's primary purpose seems to be a different one (retrying node IP retrieval from the API), so my understanding is that the check was not really needed; unless I'm misunderstanding things which I don't want to rule out at all. The quick workaround was to explicitly configure kube-proxy with . However, I'm not entirely sure if that's a good solution long-term compared to letting Kubernetes do The Right Thing, which is what I'd like to clarify through this issue. Environment: Kubernetes version (use ): v1.16.1 Cloud provider or hardware configuration: Kubernetes running on DigitalOcean VMs (droplets) OS (e.g: ): Debian GNU/Linux 9 (stretch) Kernel (e.g.
Code change:
masqueradeValue := 1 << uint(masqueradeBit) masqueradeMark := fmt.Sprintf("%#08x/%#08x", masqueradeValue, masqueradeValue) <del> if nodeIP == nil { klog.Warningf("invalid nodeIP, initializing kube-proxy with 127.0.0.1 as nodeIP") nodeIP = net.ParseIP("127.0.0.1") } </del> isIPv6 := utilnet.IsIPv6(nodeIP) klog.V(2).Infof("nodeIP: %v, isIPv6: %v", nodeIP, isIPv6)

Related discussion:
text: ): Linux master-k8s-1-16-test1 4.19.0-0.bpo.6-amd64 SMP Debian 4.19.67-2+deb10u1~bpo9+1 (2019-09-30) x86_64 GNU/Linux /sig network /cc who were involved in the referenced PR.
I made a similar comment in , wondering if this is a breaking change in v1.16 that we should revert? Previously if is nil, kube-proxy defaults to but now kube-proxy always exits with error instead of defaulting to local host. I think a lot of external cloud providers relied on this behavior because they needed to talk to API server prior to node addresses being set by the provider. Thoughts /assign
Opened to preserve behavior before v1.16 where node IP defaults to 127.0.01 is none is found. PTAL
Code change:
} func (r realJobControl) DeleteJob(namespace string, name string) error { <del> return r.KubeClient.BatchV1().Jobs(namespace).Delete(name, nil) </del> <ins> background := metav1.DeletePropagationBackground return r.KubeClient.BatchV1().Jobs(namespace).Delete(name, &metav1.DeleteOptions{PropagationPolicy: &background}) </ins> } type fakeJobControl struct {

Related discussion:
text: What happened: There seems to be a regression after 1.13 release. A Cronjob with default settings does not clean up completed pods. The job themselves are getting deleted as expected, but the pods are left behind. This seems to be related to What you expected to happen: Expected the pods for now non-existent child jobs of a cronjob to be cleaned up. This is the way it used to work prior to 1.13/ How to reproduce it (as minimally and precisely as possible): run on this manifest: Wait, more than 3 minutes, observe, that while only 3 successful jobs are present, the pods from now deleted jobs are left behind and never get deleted. Anything else we need to know?: pointed out the following relevant places in the code base: deleting a job orphans pods by default and the cronjob controller doesn't do anything different than the default when deleting Environment: Kubernetes version (use ): 1.13.0 Cloud provider or hardware configuration: ubuntu VM in vSphere OS (e.g. from /etc/os-release): 16.04.5 LTS (Xenial Xerus) Kernel (e.g. ): 4.4.0-135-generic Install tools: kubeadm <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/sig apps
Should include a known issues in 1.13.0 as well /priority critical-urgent
Lemme see if I can easily reproduce this.
/milestone v1.13
Fixes are in (master) and (1.13).