jiebi/CodeConvo huggingface dataset:
OK then, let's add a timeout and test both DROP and REJECT-- no sort of network problem should hang a component.
I think it didn't hang. It just that the kubelet was retrying too slow and sometimes it fit test timeout, sometime it didn't.
Could we reduce the timeout to make it always fit the test timeout? , Filip Grzadkowski
I guess we can. I'll try tomorrow.
It seems we have a new type of failure: It seems that kubelet on a machine where network is blocked doesn't notice new pod. After reading logs I assume that kubelet doesn't notice that the watch is broken and doesn't try to relist everything. It notices new pod only after watch timeout.
Ugh, I hope we don't have to start sending watch heartbeats.
It seems that all recent failures are with the following error: which means the reason is here: - can you please take a look or delegate?
I did some more debugging of it. I was looking into this failure: There is something strange happening here: run performTemporaryNetworkFailure: succesfully block the load from kubelet: the succesfully wait for node to become not ready: some time nodecontroller successfully removes the pod: replication controller creates a new pod: here is something I don't undertand - because of some reason, the test doesn't observe that the pod actually disappeared. - any thoughts on it?
BTW - this is exactly the same situation in:
OK - I think this is related to default 30s deletion grace period.
Yes - it seems that this is exactly the case. I will send out PR fixing this out for review.
I'm reopening it for tracking. If there won't be any failures within next few days - I will move the test back from flaky and close this issue.
Thanks for fixing, .
It seems that the main issue is fixed. However, there was one failure in the last 30 runs: in which one pod simply didn't started (and it was "preparation" for the test, not the test itself).
During next 2 days there was exactly one more failure of this test: it is exactly the same as the previous one. However, those are duplicates of So I'm going to move this test out of flaky suite.
There were two patches merged recently in the kernel that will significantly improve the IPVS proxier: The first patch fixes an issue where a packet is dropped whenever a stale conntrack entry is reused. This caused a 1s delay whenever the stale conntrack entry was used. This was the original reason for setting . More details in The second patch fixes an issue when a source port is reused from a given client and the corresponding destination was deleted. This is easily reproducible when a single client sends a lot traffic to a small set of destinations and one of the destination servers are removed. More details in Based on these two patches, we should start setting (default) starting on Linux v5.9 when those two patches make it to the main line. The new sysctls for the IPVS proxier would be: We should probably wait for v5.9 before making any changes to kube-proxy, this is just a tracking issue so we don't forget.
/sig network /area ipvs
Should probably hold on the PR until v5.9 is released later this year. cc
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry. ‍
/remove-triage unresolved
Yes it makes sense to wait until a kernel with the patch exists ;) I think we can also avoid touching it and follow the approach "only touch the sysctl we absolutely need, otherwise trust the admin to know what they need"
I install linux-5.9-rc2 in centos7. set /proc/sys/net/ipv4/vs/connreusemode = 1 i have three service: service-a ---service-b --service-c in k8s. service-c return "helloworld" I use ab to benchmark service-a. [root ~]# ab -n -c 1000 http://172.17.2.66:/ This is ApacheBench, Version 2.3 &lt;Copyright 1996 Adam Twiss, Zeus Technology Ltd, Licensed to The Apache Software Foundation, Benchmarking 172.17.2.66 (be patient) Completed 1000 requests Completed 2000 requests Completed 3000 requests Completed 4000 requests Completed 5000 requests Completed 6000 requests Completed 7000 requests Completed 8000 requests Completed 9000 requests Completed requests Finished requests Server Software: Server Hostname: 172.
17.2.66 Server Port: Document Path: / Document Length: 11 bytes Concurrency Level: 1000 Time taken for tests: 4.128 seconds Complete requests: Failed requests: 0 Write errors: 0 Total transferred: bytes HTML transferred: bytes Requests per second: 2422.76 [#/sec] (mean) Time per request: 412.753 [ms] (mean) Time per request: 0.413 [ms] (mean, across all concurrent requests) Transfer rate: 302.84 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 0 16 71.5 4 1004 Processing: 4 216 92.9 202 2340 Waiting: 4 215 92.7 202 2340 Total: 4 232 121.5 215 2341 Percentage of the requests served within a certain time (ms) 50% 215 66% 253 75% 280 80% 299 90% 348 95% 412 98% 490 99% 556 100% 2341 (longest request) [root ~]# ab -n -c 1000 http://172.17.2.66:/ This is ApacheBench, Version 2.3 &lt;Copyright 1996 Adam Twiss, Zeus Technology Ltd, Licensed to The Apache Software Foundation, Benchmarking 172.17.2.66 (be patient) Completed 1000 requests Completed 2000 requests Completed 3000 requests Completed 4000 requests Completed 5000 requests Completed 6000 requests Completed 7000 requests Completed 8000 requests Completed 9000 requests Completed requests Finished requests Server Software: Server Hostname: 172.17.2.66 Server Port: Document Path: / Document Length: 11 bytes Concurrency Level: 1000 Time taken for tests: 7.071 seconds Complete requests: Failed requests: 0 Write errors: 0 Total transferred: bytes HTML transferred: bytes Requests per second: 1414.21 [#/sec] (mean) Time per request: 707.110 [ms] (mean) Time per request: 0.707 [ms] (mean, across all concurrent requests) Transfer rate: 176.
78 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 0 9 66.2 2 1019 Processing: 10 661 575.2 372 3603 Waiting: 1 659 573.7 372 3603 Total: 10 670 584.6 380 3607 Percentage of the requests served within a certain time (ms) 50% 380 66% 742 75% 1008 80% 1183 90% 1535 95% 1784 98% 2376 99% 2479 100% 3607 (longest request) [root ~]# ab -n -c 1000 http://172.17.2.66:/ This is ApacheBench, Version 2.3 &lt;Copyright 1996 Adam Twiss, Zeus Technology Ltd, Licensed to The Apache Software Foundation, Benchmarking 172.17.2.66 (be patient) Completed 1000 requests Completed 2000 requests Completed 3000 requests Completed 4000 requests Completed 5000 requests Completed 6000 requests Completed 7000 requests Completed 8000 requests Completed 9000 requests Completed requests Finished requests Server Software: Server Hostname: 172.17.2.66 Server Port: Document Path: / Document Length: 11 bytes Concurrency Level: 1000 Time taken for tests: 13.580 seconds Complete requests: Failed requests: 0 Write errors: 0 Total transferred: bytes HTML transferred: bytes Requests per second: 736.40 [#/sec] (mean) Time per request: 1357.958 [ms] (mean) Time per request: 1.358 [ms] (mean, across all concurrent requests) Transfer rate: 92.05 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 0 4 6.6 1 49 Processing: 11 1333 491.9 1307 3185 Waiting: 1 1330 491.9 1305 3185 Total: 17 1337 491.
9 1311 3191 Percentage of the requests served within a certain time (ms) 50% 1311 66% 1565 75% 1679 80% 1748 90% 1911 95% 2129 98% 2384 99% 2650 100% 3191 (longest request) [root ~]# ab -n -c 1000 http://172.17.2.66:/ This is ApacheBench, Version 2.3 &lt;Copyright 1996 Adam Twiss, Zeus Technology Ltd, Licensed to The Apache Software Foundation, Benchmarking 172.17.2.66 (be patient) Completed 1000 requests Completed 2000 requests Completed 3000 requests Completed 4000 requests Completed 5000 requests Completed 6000 requests Completed 7000 requests Completed 8000 requests Completed 9000 requests Completed requests Finished requests Server Software: Server Hostname: 172.17.2.66 Server Port: Document Path: / Document Length: 11 bytes Concurrency Level: 1000 Time taken for tests: 23.720 seconds Complete requests: Failed requests: 936 (Connect: 0, Receive: 0, Length: 936, Exceptions: 0) Write errors: 0 Total transferred: bytes HTML transferred: bytes Requests per second: 421.58 [#/sec] (mean) Time per request: 2372.009 [ms] (mean) Time per request: 2.372 [ms] (mean, across all concurrent requests) Transfer rate: 47.77 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 0 5 7.5 1 49 Processing: 8 2345 3364.8 333 Waiting: 0 1744 3030.9 277 Total: 8 2350 3364.8 340 Percentage of the requests served within a certain time (ms) 50% 340 66% 1434 75% 3390 80% 5490 90% 8669 95% 9233 98% 99% 100% (longest request) the last time tcp connections of service-b :
/assign Sorry, but I do not understand what are you trying to say.
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --command fails because of I think the link is from k/k's Am I right?? kubetest command succeeds With installed, OR Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig release /sig testing
/assign
Need cherry-picks for 1.21, 1.20 to address this. is updated but downloaded downloads and executes its /reopen
Reopened this issue. <detailsIn response to : Need cherry-picks for 1.21, 1.20 to address this. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
cc
this was cherry picked AFAICT, thank you!
The function in accepts an parameter which it describes as a way to "controller specific flags to be included in the complete set of controller manager flags". However, when the usage string is shown (either via or when parsing options fails), these additional options are not included in the usage string. This was identified via code inspection while trying to figure out what options did in . For e.g. , the output of should include a section corresponding to . This section should in turn list all of . a cloud provider executable that includes additional options, such as the additional options are not shown in the output No response_ n/a (all cloud providers are affected) n/a n/a n/a n/a
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cloud-provider
<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? No What keywords did you search in Kubernetes issues before filing this one? static manifest Is this a BUG REPORT or FEATURE REQUEST? (choose one): Bug report Kubernetes version (use ): Environment: Cloud provider or hardware configuration: Bare metal and reproduced in Vagrant OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: custom scratch install and reproduced with kubeadm Others: Calico for networking. Used (Also able to reproduce without calico) What happened: Modified one of the static manifest files in . The kubelet killed the running pod and did NOT create a new pod. (waited 30 minutes, still no pod) What you expected to happen: Kubelet should have killed the pod and started a new one. How to reproduce it (as minimally and precisely as possible): Started with single node, vanilla Ubuntu Xenial, installed kubernetes via kubeadm, nothing special. Modify , for example change the logging level or even just whitespace at the end of the file. Observe the kubelet kill the controller manager pod, and does not restart it. Anything else we need to know: If I remove the file from the directory, and then replace the file, kubelet will start it up. Kubelet logs from the period where I updated the file:
Did you modify the yaml file using an editor (e.g., vim)? If the editor creates a temporary file in the same directory, kubelet can get confused and thinking that's a new pod (with the same name & namespace). When the temporary file gets removed, kubelet'd inadvertently delete the pod from its cache. Since kubelet relies on inotify for changes, if it doesn't get notified for any new changes from the original yaml file, it'd forever ignore the pod. There is an issue open to add periodic listing in addition to inotify (introduced in 1.5). This would make sure kubelet eventually recover from the situation. I can't find the issue, but there is another one for inotify: . Another fix is to make sure kubelet ignore such known temporary files (it already ignores files starting with ). In general, don't edit the file directly.
Copy the file to another place, edit, and copy it back to the manifests directory.
I'm seeing a similar problem except I never edited the file, might be fit for a separate issue
Yes I was using vim, and this explanation makes sense. Would be nice if it was smart about it. Would a PR to ignore the vim temp file be acceptable
I think it's reasonable. I doubt if anyone uses those file names on purpose.
The dot ignore should have covered the vim temp file too, but it doesn't work because the filename passed in is the absolute filename including the path. So the first character is not a dot. I'll try to submit a PR fix.
PR submitted
fixed?
How to work around? I coudn't start api server when using kubeadm to deploy k8s. Thanks!
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: Not able to change the default storage class. What you expected to happen: Default storage class to change. How to reproduce it (as minimally and precisely as possible): But after few mins. Anything else we need to know?: Tried to remove the annotation but no luck. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: minikube OS (e.g. from /etc/os-release): Buildroot 2018.05 Kernel (e.g. ): 4.16.14 Install tools: minikube Others:
/assign
Those labels are not set on the issue: <detailsIn response to : /remove-sig api-machinery Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cluster-lifecycle
The cause is that kube-addon-manager is reconciling the standard storage class.
Those labels are not set on the issue: <detailsIn response to : /remove-sig api-machinery Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Thanks for looking into it. Sorry for native question, I want to keep only ceph as a default storage class how can I do that?
hi I'm not really sure about the actual command to fix this. Basically, you could log into the VM and do the same modification as the PR above does, then restart minikube.
Has the fix made it into minikube yet? I'm still seeing the issue in 0.31.0
Looking at PR , the fix was made in kubernetes core, not minikube. The error still exists in minikube:
<pre:::: fang 10:04:02 (cgthayer-pchat) 0 platform; kubectl top nodes NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-charles-test-cluster-default-pool--jkrj 311m 33% 2470Mi 93% gke-charles-test-cluster-default-pool--krsh 154m 16% 1763Mi 66% gke-charles-test-cluster-default-pool--vhgq 265m 28% 1954Mi 73% gke-charles-test-cluster-default-pool--x3bs 186m 19% 1990Mi 75% gke-charles-test-cluster-default-pool--073f 270m 28% 1729Mi 65% gke-charles-test-cluster-default-pool--09zf 403m 42% 2698Mi 102% gke-charles-test-cluster-default-pool--p7h1 191m 20% 1582Mi 59% :::: fang 10:04:07 (cgthayer-pchat) 0 platform; kubectl get nodes NAME STATUS ROLES AGE VERSION gke-charles-test-cluster-default-pool--073f Ready <none7d v1.7.12-gke.1 gke-charles-test-cluster-default-pool--09zf Ready <none7d v1.7.12-gke.1 gke-charles-test-cluster-default-pool--cl65 Ready <none6d v1.7.12-gke.1 gke-charles-test-cluster-default-pool--jkrj Ready <none8d v1.7.12-gke.1 gke-charles-test-cluster-default-pool--krsh Ready <none8d v1.7.12-gke.1 gke-charles-test-cluster-default-pool--p7h1 Ready <none7d v1.7.12-gke.1 gke-charles-test-cluster-default-pool--vhgq Ready <none8d v1.7.12-gke.1 gke-charles-test-cluster-default-pool--x3bs Ready <none8d v1.7.12-gke.1 <preWhat you expected to happen: I expected all nodes to be printed.
Since the missing node might have bad data I might expect it to have the same number of columns with "-" values and a note in a new 6th column saying something like "unavailabe" "unresponsive" "stale-data" etc. e.g. <pregke-charles-test-cluster-default-pool--p7h1 - - - - unreachable </preHow to reproduce it (as minimally and precisely as possible): Not sure. Perhaps run a pod where you can intentionally trigger it to use too much cpu or memory. Anything else we need to know?: Environment: Kubernetes version (use ): kubectl version Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.6", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-12-21T06:34:11Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"8+", GitVersion:"v1.8.8-gke.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-02-16T18:26:58Z", GoVersion:"go1.8.3b4", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: GKE OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig cli
/kind feature
I raise a PR to print when the metrics is missing. As for the 6th column you mentioned, I think it could be debatable for its necessity. :)
See Jenkins job kubernetes-upgrade-1.0-1.1-gke-step3-e2e-old's last few runs. The relevant line is . Note that this is running e2e tests off of the branch. It's possible that this is also caused by , but it's not clear to me.
This is also failing on kubernetes-upgrade-1.0-1.1-gke-step5-e2e-old, where we're running tests against a cluster that's been fully upgraded, (master and nodes,) from 1.0 to 1.1.
Status update: this may be caused by , so that issue should be prioritized, but it's not totally clear if that will fix this problem.
said he would take a look and update the issue with any findings. It may be a bit premature to look into it before is resolved, but at least we can try to figure out how to manually repro this now, so that we will know whether fixes this issue.
I'm unable to find logs from failed runs, which makes everything slightly harder. I'll try to reproduce the problem on GCE cluster, but it may not work. FYI: most of was caused by waiting for skydns to come back up (which doesn't happen: ). I see that there's a difference in number of system pods between version 1.1 and 1.0 - there's only 5 of them in 1.1 and 8 of them in 1.0. I'm wondering if this may cause problems in 'old' version of test which we're running there. I'm going to try reproduce it on GCE (as this is the setup I understand better) and if I'll fail I'll try again on GKE.
AFAICT there's a problem with the test, not the upgrade itself. The test from 1.1 branch fails on 1.0 cluster.
It's caused by . Can we please stop renaming things? AFAICT the problem is with test only, everything should be working 'on production'. In v1.0 scheduler Events reasons start with a small letter (e.g. failedScheduling) and v1.0 test is looking for those kind of events to occur. In v1.1 we changed Event reasons to start with a capital letter (e.g.
FailedScheduling) and modified test accordingly. Effects are obvious. We can: revert patch it to v1.0 merge and patch a test fix which adds handling both versions () I'm for the last option, but whatever we do (except reverting) will break backward compatibility for everyone depending/counting/filtering/whatnot on scheduler Event reasons
thanks for your work on this. From on : Furthermore, in my (rookie) opinion, seems like it should be considered a bug fix rather than an API change. Thus I think makes good sense.
Thanks a lot for debugging this, and I agree we should go with your PR. I talked to Brian and he said we don't consider changes to values of fields in events to be breaking API changes. Unfortunately people do seem to depend on these in various places. One solution is what I proposed in so the compiler could prevent these incompatibilities (at least for components and tests that are compiled together).
+1 to creating a file containing constants for the event reasons that are currently in use.
I'll prepare a cherry pick.
Reopening until cherry pick is merged.
Cherry pick is .
The test mentioned in the first entry in this thread is still failing but not due to the error message related to this issue. Since the cherrypick has been merged, can we close this issue, and followup with additional debugging in ?
Works for me. Thanks to again for your work on this.
I actually still see failures of the same kind in (master 1.1, nodes 1.0) and (master 1.1, nodes 1.1). I assume this is because we haven't cherrypicked test fix into . So this is now a duplicate of .
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: What you expected to happen: kube-controller-manager should not panic. How to reproduce it (as minimally and precisely as possible): Kubernetes 1.12.6, , and Anything else we need to know?: It's running and , and logs says: Don't see any notable errors in . Is that because it's running the old ? Basically, is expecting , but somehow it's getting which I assume Environment: Kubernetes version (use ): v1.12.6 Cloud provider or hardware configuration: AWS EC2 OS (e.g: ): Amazon Linux 2 Kernel (e.g. ): 4.14.109-99.92.amzn2.x86_64
Could be related to , but this happens with all the patches from the issue /sig scalability /cc
/sig autoscaling /assign can you triage please?
/sig-remove scalabilty
/sig remove scalability
/sig scalability remove
/remove-sig scalability
API returned v1.Status instead of v1beta2.MetricValueList. We should check what is the type of the object
/sig instrumention
/assign
/sig instrumentation
seems promising. Is it missing anything?
I've root-caused the issue. Explained here -
Yep, should address the issue.
<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: N/A Which test(s) are failing: should function for node-pod communication: udp Since when has it been failing: Unknown. Testgrid link: The test is currently excluded. Reason for failure: By default, cmd is used to execute commands in Windows containers, but it has some issues executing the command (no output): echo 'hostName' | nc -w 1 -u <targetIP<targetPortBut it has no issues running (output is correct): echo hostName | nc -w 1 -u <targetIP<targetPortThis causes the test "should function for node-pod communication: udp" to fail when using Windows Containers. Anything else we need to know: /sig testing /sig windows