jiebi/CodeConvo huggingface dataset:
Code change:
These tests have flaked consistently since merged: in many cases, they are the only failure in a test run: Either the functionality introduced by that PR is not reliable, or the test is faulty. /sig node /priority critical-urgent /assign
/milestone v1.19
/assign
Looks like this is failing ~5% of pull-kubernetes-node-e2e runs
FYI
Looks like there might still be a race somewhere looking at the kubelet logs Success case: Fail case:
Looking at it more, it seems like is unexpectedly empty here Success: Fail:
Stacks of interest I think we might be dealing with a stale If there is a call before the container is to the Pod in the runtime and the is called within the 2 second default cache period, it seems possible that the in the could have no containers.
what do you think of this analysis? If so, any obvious solutions occur to you?
If I'm correct, didn't introduce this; just the e2e test exposed an existing bug in the kubelet.
That all looks correct. I'm taking a look at the proposed solutions

Related discussion:
text: <ins> /* Copyright 2020 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package kubelet import ( "testing" "time" "github.com/golang/groupcache/lru" ) func TestTimeCache(t *testing.T) { cache := &timeCache{cache: lru.New(2)} if a, ok := cache.Get("123"); ok { t.Errorf("expected cache miss, got %v, %v", a, ok) } now := time.Now() soon := now.Add(time.Minute) cache.Add("now", now) cache.Add("soon", soon) if a, ok := cache.Get("now"); !ok || !a.Equal(now) { t.Errorf("expected cache hit matching %v, got %v, %v", now, a, ok) } if a, ok := cache.Get("soon"); !ok || !a.Equal(soon) { t.Errorf("expected cache hit matching %v, got %v, %v", soon, a, ok) } then := now.Add(-time.Minute) cache.Add("then", then) if a, ok := cache.Get("now"); ok { t.Errorf("expected cache miss from oldest evicted value, got %v, %v", a, ok) } if a, ok := cache.Get("soon"); !ok || !a.Equal(soon) { t.Errorf("expected cache hit matching %v, got %v, %v", soon, a, ok) } if a, ok := cache.Get("then"); !ok || !a.Equal(then) { t.Errorf("expected cache hit matching %v, got %v, %v", then, a, ok) } } </ins>
Code change:
Which jobs are failing: ci-kubernetes-node-kubelet-conformance Which test(s) are failing: Variable Expansion should succeed in writing subpaths in container Variable Expansion should verify that a failing subpath expansion can be modified during the lifecycle of a container Since when has it been failing: Since Fri Jul 10 15:41:46 2020 -0700, when was merged Testgrid link: https://k8s- Reason for failure: Used DeletePropagationForeground option to delete test pod introduced by Anything else we need to know: The test pod stuck in Terminating state until 5m timeout of DeletePodWithWait expired. It happens with empty pod as well. Here is a simplified ginkgo test case that triggers this failure: The pod phase is "Failed" and container reason is "Error" for some reason: Reverting above commit or changing delete option to DeletePropagationBackground should fix this.
/cc
/sig node
/sig storage cc
/assign
I think change just triggered the underlying issue. DeletePropagationForeground option doesn't work as expected it seems.
does the ci-kubernetes-node-kubelet-conformance job run the controller-manager? the kube-controller-manager is responsible for deleting objects with a foreground finalizer on them, but I don't see any kube-controller-manager logs gathered in the artifacts for one of those test failures -
I could reproduce the issue for commit (before 'generic ephemeral inline volumes' PR was merged) + 'use foreground deletion' commit.
yeah, I tend to think switching that e2e helper method for all callers was not a good idea
it looks like node e2e runs only etcd, api server and namespace controller: I don't think it makes sense to run controller-manager just to fix this failure. Would modifying test framework to set DeletePropagationBackground option for e2e node tests be correct solution?
For a quick fix, I would suggest just reverting the test change for now. Long term, yes, it should be configurable per test.
Also do we have plans to make this job a blocking presubmit test so we can find these problems earlier?
I agree, we should revert , both to avoid changing behavior for existing callers and to match how pods are normally deleted
if we need a PR for that I can prepare it.
that would be great, thanks
putting my hat on, probably not? more hour+ tests that run on every PR are a non-goal
The SIG / OWNERs should enable testgrid alerts on failures, if periodic testing is not sufficient you can use a postsubmit to run whenever we merge code. We're not looking to rely further on presubmit, as is we're running too much there leading to excessive resource usage and inability to merge fixes like this in a timely manner between one thing or the other flaking heavily (and our odds get much worse with each new presubmit)

Related discussion:
text: } // DeletePodWithWait deletes the passed-in pod and waits for the pod to be terminated. Resilient to the pod <del> // not existing. Also waits for all owned resources to be deleted. </del> <ins> // not existing. </ins> func DeletePodWithWait(c clientset.Interface, pod *v1.Pod) error { if pod == nil { return nil
Code change:
<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: gce-master-scale-correctness Which test(s) are flaking: [sig-network] Ingress API should support creating Ingress API operations [Conformance] Testgrid link: https://k8s- Reason for failure: Anything else we need to know: /kind flake /sig scalability /sig network /cc
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry. ‍
this is caused by the issue described in a different test is creating a default ingressclass, which breaks creation of ingress objects if there was already a default ingressclass set up in the cluster /assign
this should resolve the issue:
I take it back, looks like the conformance test is also creating default ingressclass objects... it should not fix in

Related discussion:
text: cs = f.ClientSet }) <del> ginkgo.It("should set default value on new IngressClass", func() { </del> <ins> ginkgo.It("should set default value on new IngressClass [Serial]", func() { </ins> ingressClass1, err := createIngressClass(cs, "ingressclass1", true, f.UniqueName) framework.ExpectNoError(err) defer deleteIngressClass(cs, ingressClass1.Name)
Code change:
<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: gce-master-scale-correctness Which test(s) are flaking: [sig-network] Ingress API should support creating Ingress API operations [Conformance] Testgrid link: https://k8s- Reason for failure: Anything else we need to know: /kind flake /sig scalability /sig network /cc
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry. ‍
this is caused by the issue described in a different test is creating a default ingressclass, which breaks creation of ingress objects if there was already a default ingressclass set up in the cluster /assign
this should resolve the issue:
I take it back, looks like the conformance test is also creating default ingressclass objects... it should not fix in

Related discussion:
text: } }) <del> ginkgo.It("should not set default value if no default IngressClass", func() { </del> <ins> ginkgo.It("should not set default value if no default IngressClass [Serial]", func() { </ins> ingressClass1, err := createIngressClass(cs, "ingressclass1", false, f.UniqueName) framework.ExpectNoError(err) defer deleteIngressClass(cs, ingressClass1.Name)
Code change:
<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: gce-master-scale-correctness Which test(s) are flaking: [sig-network] Ingress API should support creating Ingress API operations [Conformance] Testgrid link: https://k8s- Reason for failure: Anything else we need to know: /kind flake /sig scalability /sig network /cc
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry. ‍
this is caused by the issue described in a different test is creating a default ingressclass, which breaks creation of ingress objects if there was already a default ingressclass set up in the cluster /assign
this should resolve the issue:
I take it back, looks like the conformance test is also creating default ingressclass objects... it should not fix in

Related discussion:
text: } }) <del> ginkgo.It("should prevent Ingress creation if more than 1 IngressClass marked as default", func() { </del> <ins> ginkgo.It("should prevent Ingress creation if more than 1 IngressClass marked as default [Serial]", func() { </ins> ingressClass1, err := createIngressClass(cs, "ingressclass1", true, f.UniqueName) framework.ExpectNoError(err) defer deleteIngressClass(cs, ingressClass1.Name)
Code change:
<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: gce-master-scale-correctness Which test(s) are flaking: [sig-network] Ingress API should support creating Ingress API operations [Conformance] Testgrid link: https://k8s- Reason for failure: Anything else we need to know: /kind flake /sig scalability /sig network /cc
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry. ‍
this is caused by the issue described in a different test is creating a default ingressclass, which breaks creation of ingress objects if there was already a default ingressclass set up in the cluster /assign
this should resolve the issue:
I take it back, looks like the conformance test is also creating default ingressclass objects... it should not fix in

Related discussion:
text: // IngressClass resource create/read/update/watch verbs ginkgo.By("creating") <del> ingressClass1, err := createIngressClass(cs, "ingressclass1", true, f.UniqueName) </del> <ins> ingressClass1, err := createIngressClass(cs, "ingressclass1", false, f.UniqueName) </ins> framework.ExpectNoError(err) <del> _, err = createIngressClass(cs, "ingressclass2", true, f.UniqueName) </del> <ins> _, err = createIngressClass(cs, "ingressclass2", false, f.UniqueName) </ins> framework.ExpectNoError(err) <del> _, err = createIngressClass(cs, "ingressclass3", true, f.UniqueName) </del> <ins> _, err = createIngressClass(cs, "ingressclass3", false, f.UniqueName) </ins> framework.ExpectNoError(err) ginkgo.By("getting")
Code change:
The statement "returns [...], or nil" cannot be true since the return type is a struct: Suggest it could say ", or an empty struct if ..." /sig api-machinery
I can file a PR to fix the incorrect comment if you don't mind :)
Sure, I didn't get round to it yet.
/assign

Related discussion:
text: // SetGroupVersionKind sets or clears the intended serialized kind of an object. Passing kind nil // should clear the current setting. SetGroupVersionKind(kind GroupVersionKind) <del> // GroupVersionKind returns the stored group, version, and kind of an object, or nil if the object does // not expose or provide these fields. </del> <ins> // GroupVersionKind returns the stored group, version, and kind of an object, or an empty struct // if the object does not expose or provide these fields. </ins> GroupVersionKind() GroupVersionKind }
Code change:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: not fixed the SMB mount issue totally, when there is a SMB mount with same remote path, original SMB mapping would be removed and SMB mapping re-established, that would cause temporally data inaccessible with the original container. should change to should use to check whether original smb mapping is valid, here is an example showing that after SMB server password changed, could validate whether connection is ok: the main diff on smb mount between linux and windows is that on LInux, every smb mount is independent, while on Windows, one remote smb server address(e.g. ) can only have one SMB-GlobalMapping. So on Windows, there is a difficulty: in some cases, we should skip create SMB-GlobalMapping since it already exists in other cases, original password is incorrect or expired, user wants to delete SMB-GlobalMapping, and then create a new SMB-GlobalMapping (fixed by ) On Windows, there is no way to check whether original SMB-GlobalMapping is valid or not, that was the answer about 1 years ago from windows team, will update this thread(also about the max SMB-GlobalMapping limit) if there is any new update recently. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig storage /assign

Related discussion:
text: "port": 10001, "labels": { "name": "redisslave" <del> } </del> <ins> }, </ins> "selector": { "name": "redisslave" }
Code change:
What happened: I two CSI plugins to my Kubernetes cluster via a single call to . When I did this, the annotation was improperly set on 2 out of 3 nodes. Only one of the plugins appeared: What you expected to happen: Both and plugins should have appeared in the annotation. How to reproduce it (as minimally and precisely as possible): Apply two or more CSI plugins via a single call to . Anything else we need to know?: Kubelet logs indicate that it registered both plugins at the same time: The CSINode object looks complete: Only the node's annotation did not include Restarting kubelet fixed the issue. It looks like a race condition when InstallCSIDriver is called concurrently for multiple drivers. The code here: could lead to each thread observing the annotation as being empty, and then setting its own annotation that includes only itself. Environment: Kubernetes version (use ): v1.17.11 Cloud provider or hardware configuration: AWS OS (e.g: ): Ubuntu 20.04.1 LTS Kernel (e.g. ): Linux ip-172-31-17-163 5.4.0-1021-aws -Ubuntu SMP Fri Jul 24 09:42:29 UTC 2020 x8664 x8664 x86_64 GNU/Linux Install tools: juju
/sig storage
/assign
I can reproduce this issue with commands below ! The correct csi nodeid would be like this

Related discussion:
text: "fmt" "math" "strings" <ins> "sync" </ins> "time"
Code change:
What happened: I two CSI plugins to my Kubernetes cluster via a single call to . When I did this, the annotation was improperly set on 2 out of 3 nodes. Only one of the plugins appeared: What you expected to happen: Both and plugins should have appeared in the annotation. How to reproduce it (as minimally and precisely as possible): Apply two or more CSI plugins via a single call to . Anything else we need to know?: Kubelet logs indicate that it registered both plugins at the same time: The CSINode object looks complete: Only the node's annotation did not include Restarting kubelet fixed the issue. It looks like a race condition when InstallCSIDriver is called concurrently for multiple drivers. The code here: could lead to each thread observing the annotation as being empty, and then setting its own annotation that includes only itself. Environment: Kubernetes version (use ): v1.17.11 Cloud provider or hardware configuration: AWS OS (e.g: ): Ubuntu 20.04.1 LTS Kernel (e.g. ): Linux ip-172-31-17-163 5.4.0-1021-aws -Ubuntu SMP Fri Jul 24 09:42:29 UTC 2020 x8664 x8664 x86_64 GNU/Linux Install tools: juju
/sig storage
/assign
I can reproduce this issue with commands below ! The correct csi nodeid would be like this

Related discussion:
text: nodeName        types.NodeName volumeHost      volume.VolumeHost migratedPlugins map[string](func() bool) <ins> // lock protects changes to node. lock sync.Mutex </ins> } // If no updates is needed, the function must return the same Node object as the input.
Code change:
What happened: I two CSI plugins to my Kubernetes cluster via a single call to . When I did this, the annotation was improperly set on 2 out of 3 nodes. Only one of the plugins appeared: What you expected to happen: Both and plugins should have appeared in the annotation. How to reproduce it (as minimally and precisely as possible): Apply two or more CSI plugins via a single call to . Anything else we need to know?: Kubelet logs indicate that it registered both plugins at the same time: The CSINode object looks complete: Only the node's annotation did not include Restarting kubelet fixed the issue. It looks like a race condition when InstallCSIDriver is called concurrently for multiple drivers. The code here: could lead to each thread observing the annotation as being empty, and then setting its own annotation that includes only itself. Environment: Kubernetes version (use ): v1.17.11 Cloud provider or hardware configuration: AWS OS (e.g: ): Ubuntu 20.04.1 LTS Kernel (e.g. ): Linux ip-172-31-17-163 5.4.0-1021-aws -Ubuntu SMP Fri Jul 24 09:42:29 UTC 2020 x8664 x8664 x86_64 GNU/Linux Install tools: juju
/sig storage
/assign
I can reproduce this issue with commands below ! The correct csi nodeid would be like this

Related discussion:
text: // the effects of previous updateFuncs to avoid potential conflicts. For example, if multiple // functions update the same field, updates in the last function are persisted. func (nim *nodeInfoManager) tryUpdateNode(updateFuncs ...nodeUpdateFunc) error { <ins> nim.lock.Lock() defer nim.lock.Unlock() </ins> // Retrieve the latest version of Node before attempting update, so that // existing changes are not overwritten.
Code change:
What happened: I download fluentd-elasticsearch in addons. The following error occurs then I use "kubectl apply -f es- " Then, i find the elasticsearch-logging pod restart lots of time, and keep status with CrashLoopBackOff. [root ~]# kubectl logs -f elasticsearch-logging-0 -n kube-system export NODENAME=elasticsearch-logging-0 NODENAME=elasticsearch-logging-0 export NODEMASTER=true NODEMASTER=true export NODEDATA=true NODEDATA=true export HTTPPORT=9200 HTTPPORT=9200 export TRANSPORTPORT=9300 TRANSPORTPORT=9300 export MINIMUMMASTERNODES=2 MINIMUMMASTERNODES=2 chown -R elasticsearch:elasticsearch /data ./bin/elasticsearchloggingdiscovery I0902 14:25:06. 9 ] Kubernetes Elasticsearch logging discovery I0902 14:25:06. 9 ] Found [] I0902 14:25:16. 9 ] Found [] I0902 14:25:26. 9 ] Found [] I0902 14:25:36. 9 ] Found [] I0902 14:25:46. 9 ] Found [] I0902 14:25:56. 9 ] Found []
unclear whether the addon is up to date and supported at this point. /sig instrumentation
cc
/assign I will take a look this week.
I have the same problem k8s version:v1.18.8
same
I've pushed a change to the stateful set that should unblock people as there was a 1.18 compatibility problem. however, please make sure you are also creating the service via , the discovery program finds the other elasticsearch nodes via service registration.
please feel free to re-open this issue if the latest version doesn't work for you.
Why was this PR to v1.20?
probably because that is the next release. most people who use this addon do so from master. Do you need this packaged in some other way?

Related discussion:
text: addonmanager.kubernetes.io/mode: Reconcile kubernetes.io/name: "Elasticsearch" spec: <ins> clusterIP: None </ins> ports: <del> - port: 9200 protocol: TCP targetPort: db </del> <ins> - name: db port: 9200 protocol: TCP targetPort: 9200 - name: transport port: 9300 protocol: TCP targetPort: 9300 publishNotReadyAddresses: true </ins> selector: k8s-app: elasticsearch-logging <ins> sessionAffinity: None type: ClusterIP </ins>
Code change:
What happened: I download fluentd-elasticsearch in addons. The following error occurs then I use "kubectl apply -f es- " Then, i find the elasticsearch-logging pod restart lots of time, and keep status with CrashLoopBackOff. [root ~]# kubectl logs -f elasticsearch-logging-0 -n kube-system export NODENAME=elasticsearch-logging-0 NODENAME=elasticsearch-logging-0 export NODEMASTER=true NODEMASTER=true export NODEDATA=true NODEDATA=true export HTTPPORT=9200 HTTPPORT=9200 export TRANSPORTPORT=9300 TRANSPORTPORT=9300 export MINIMUMMASTERNODES=2 MINIMUMMASTERNODES=2 chown -R elasticsearch:elasticsearch /data ./bin/elasticsearchloggingdiscovery I0902 14:25:06. 9 ] Kubernetes Elasticsearch logging discovery I0902 14:25:06. 9 ] Found [] I0902 14:25:16. 9 ] Found [] I0902 14:25:26. 9 ] Found [] I0902 14:25:36. 9 ] Found [] I0902 14:25:46. 9 ] Found [] I0902 14:25:56. 9 ] Found []
unclear whether the addon is up to date and supported at this point. /sig instrumentation
cc
/assign I will take a look this week.
I have the same problem k8s version:v1.18.8
same
I've pushed a change to the stateful set that should unblock people as there was a 1.18 compatibility problem. however, please make sure you are also creating the service via , the discovery program finds the other elasticsearch nodes via service registration.
please feel free to re-open this issue if the latest version doesn't work for you.
Why was this PR to v1.20?
probably because that is the next release. most people who use this addon do so from master. Do you need this packaged in some other way?

Related discussion:
text: k8s-app: elasticsearch-logging addonmanager.kubernetes.io/mode: Reconcile rules: <del> - apiGroups: - "" resources: - "services" - "namespaces" - "endpoints" verbs: - "get" </del> <ins> - apiGroups: - "" - resources: - "services" - "namespaces" - "endpoints" - verbs: - "get" </ins> --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1