jiebi/CodeConvo huggingface dataset:
Salt was failing to install on the master vagrant instance. The change noted by the FIXME solved it.
It's possible the required merge in the upstream repo occurred, will take a look.
I just attempted to install a new cluster from upstream/master and was not able to reproduce the issue described. Was there an issue with curl-ing that specific version of salt-bootstrap? I guess more details on what the failure you encountered appeared as would help know how to proceed.
I will submit a PR to revert back to using the default non-versioned bootstrap URL. Salt-stack PR: Fixed the errors in install that were previously encountered around salt-api not being found.
See , thanks!
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: If you have a JobSpec and set with in 1.9.x, then the job will start a pod and never restart it regardless of success of failure. In 1.10.x, a job spec with and , k8s does not start any pods at all. We had to set in order to get the pod to run. The wording in the docs suggest that should only take effect when the pod is finished/or failed. It shouldn't cap starting the initial pod. How to reproduce it (as minimally and precisely as possible): Run the following job spec in 1.9.x and 1.10.x: Anything else we need to know?: The docs links this existing ticket saying is ineffective with , but the ticket talks about restarting too many times, not never starting a pod. Given this issue was slightly different, I originally did not think it was a backoffPolicy bug. Environment: Kubernetes version (use ): Tested on 1.9.7 and 1.10.6 Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): Fedora 28
/sig apps
We've been hit by this too, in v1.10.6. It seems to me that with the current behavior, a value of doesn't really make any semantic sense. I think the problem comes from the function in : called here: if is , then will end with .
sig-node : DRA related tests are failing and 2 other DRA related tests. Both jobs started failing on 7th June As per the logs DRA plugin is not getting registered to kubelet Jobs started failing after merging As per , CDI is not available out of the box for containerd which may be the reason for failure of the jobs. /sig node
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
/cc
Looks like I shouldn't have labeled DRA Node tests as NodeFeature. However, re-labeling it NodeAlphaFeature would not work either - would start to fail. I'm going to remove . This should help.
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Not fixed yet, pending
This flakiness is reproducible on x86 and ppc64le architectures. https://prow.ppc64le- in /sig
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig testing
/sig node /assign
Similar to this is to test the first contributor experience fails on ubuntu 16.04, docker version 1.12.3:
Got the same error on Linux ubuntu 4.2.0-27-generic ~14.04.1-Ubuntu SMP Fri Jan 22 15:32:26 UTC 2016 x8664 x8664 x86_64 GNU/Linux Docker Client version: 1.6.2 Client API version: 1.18 Go version (client): go1.2.1 Git commit (client): OS/Arch (client): linux/amd64 Server version: 1.6.2 Server API version: 1.18 Go version (server): go1.2.1 Git commit (server): OS/Arch (server): linux/amd64
What git sha of were you building from?
I have cloned the head. I think the reason might be missing package: gccgo-4.8-aarch64-linux-gnu I'm compiling right now and update on result.
were you running or ? runs inside docker, so the gcc error is confusing.
I run: sudo make release in Ubuntu VM running in VMWare workstation if that matters. Are you in kubernetes-users slack channel?
does fail in the same way?
Do you need to run , instead of just ? Can you try running the following:
well, if I don't do sudo docker does not work :) I need to run as root. I'll try the commands as soon as quick-release ends.
ah, gotcha. you can probably fix the docker access issue by following in the docker manual. that's probably not why it's failing, though.
make release still does not work :(
can you please run and then paste the results? ( turns up the debugging verbosity.)
for me: works (using your PR to fixes the gobindata dep) -local Go build- works (docker build) works if you have Docker running. does not work: on OSX did not seem to give more verbosity.
V=4 make release did not help :(
oops, that was not the correct environment variable, sorry. I meant .
here we go:
Aha, I think I figured it out. Sorry for making you rebuild so many times; the issue still isn't super obvious from the logs.
Basically, the bug only occurs if you don't have enough memory to do the docker builds in parallel; if that happens, you fall into this . On the parallel side of the branch, we set the platform env vars within a subshell, so each of the cross-compilation builds won't affect each other. On the non-parallel side of the branch, we don't use a subshell. Inside , there's no overrides done for . As a result, we end up incorrectly reusing the value for , , which doesn't recognize the flag being applied for the darwin build. The fix should be easy; I'll get that sent out ASAP. we probably want this fix for 1.5, since if you fall back to non-parallel crossbuilds we're doing the wrong thing.
Ack. Will keep an eye out for the fix. Please mark it as cherrypick-candidate
Thank you guys!
I think it is almost there. Got much further in the build (4 GB of RAM). Towards the end there is a failed test that makes the build fail: This is probably a different bug, as your PR got passed the misconfigured compilation.
FWIW, the worked on ubuntu 16.06. It did ran some tests apparently but the build succeeded.
I just tried with and it worked fine
since we have a PR ready and working, can we please switch this to non-release-blocker?
Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: unit tests failed. What you expected to happen: How to reproduce it (as minimally and precisely as possible): It will get . I found this issue only happened when the is truely existent in the host, Anything else we need to know?: Environment: Kubernetes version (use kubectl version): head Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. uname -a): Install tools: Others: go version go1.9.2 linux/amd64
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the and . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/area kubeadm
<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: pull-kubernetes-integration Which test(s) are failing: TestUnreservePlugin Since when has it been failing: It happens frequently this week Testgrid link: Reason for failure: I0619 11:55:31. ] PUT /api/v1/namespaces/default/endpoints/kubernetes: (2.062872ms) 200 [ (linux/amd64) kubernetes/$Format 127.0.0.1:] test : Expected the unreserve plugin to be called 2 times, was called 1 times. Anything else we need to know:
This seems to be a issue in scheduler integration tests /sig scheduling
I was analyzing that failure based on the logs from given link. Based on my analysis the problem lies in here We wait for the pod to being unschedulable, and then we compare the counts of pre-bind and unreserve called. But this is done asynchronously so it can happen that between and us reading the counts the pod has already been scheduled and pre-bind was called while unreserve was not yet called. A simple solution would be to assert instead of I'll try to submit PR with that fix to see what people think about that :)
FYI: /assign /kind flake
cc/
/close
We seemed to have curbed the kubelet memory issues in , but the runtime issues remain. https://k8s- has a lot of these. This can likely be de-duped to an auto-filed issue when the soak tests are to the munger.
I created a cluster and ran a scrip to repeatedly create/delete pods overnight, but couldn't reproduce this issue. Uploaded PR to enable the debug mode in docker, so that one can run pprof to get more useful information from the soak cluster.
I redeployed the GCE soak tests yesterday to pick up to help you debug faster. I verified that docker is running with this flag on the nodes. So as soon as it re-occurs on https://k8s- , you should be able to debug it. Just out of curiosity, what sort of debugging does that flag enable?
I know it at least enables the /debug/pprof endpoint. EDIT: and thanks for deploying a new cluster.
The new cluster started failing again: Some information I gathered: The goroutines: Will check later(?) to see if the memory uasge keeps increasing
The heap profile shows that is slowly accumulating more memory. This can be reproduced by repeatedly creating and deleting containers for thousands of times using docker v1.11 (without kubernetes involved at all). A potentially related issue is found on I didn't try it on docker v1.12 and am not sure if it suffers from the same symptom. I was able to also reproduce this through unit tests in the package. On such unit test is I modified the test to use UUIDs as keys and change the order to always insert a key and then delete right away. After running the test for 10 million iterations, the test always fails with an increase of memory usage on the heap. I filed an issue to see if the unit test failure is related, or just a red herring:
Is this p0 flake a symptom of this issue?
In , both kubelet and runtime exceed the memory limit. This issue is for the "runtime" memory issue, but not for kubelet. The kubelet memory issue seems to be relatively new. I'll look into that as well.
Update: There is a fix posted in W can't patch docker 1.11 anyway, so the soak test will continue failing.
this is the docker memory leak I mentioned yesterday.
I though this was a slow leak that only shows up in soak but apparently integration tests are hitting it too: , or is that something else, or just a confusing error message?
fwiw the logs from this failure are https://k8s-, I haven't trawled through them yet.
Looks to me like the go runtime was out of memory when building/running testmain Not sure how this is related to docker
[FLAKE-PING] This flaky-test issue would love to have more attention...
[FLAKE-PING] This flaky-test issue would love to have more attention...
This is still an issue, and would like to get this passing in the kubernetes-soak-continuous-e2e-gce-1.4 test suite for the release. I know we talked about patching docker at least for the GCE images. Is that still an option? If not, could you sync up with about how to proceed? Thanks.
Patching docker for today's GCE images is not an option. I am considering patching docker in GCI image though. cc/
It's already patched in GCI.
given that patching docker in the containervm image is not an option, the only way to pass this test is to disable checking the docker memory usage completely in the test. I'll let determine what's the best course for the release.
Here are the plan I wrote in bot auto-filed flake issue: , : was merged to stop the bleeding. I will leave this open for rest work.
These tests are now also failing in case of kubernetes-e2e-gce-gci-serial-release-1.4.
thanks for checking the last failure. Yes, this is known issue, and shouldn't block 1.4 release since GCI default on node is decoupled from the releases.
If it's not a release blocker, do you suggest that we disable the test in the 1.4 branch? cc
Filed to disable the tests for 1.4.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
To help clean up the list of flakes for 1.6, I am closing this issue as it does not appear to flake anymore with the increased limits.
kube-proxy contains a comment that "all flags below are deprecated", but many non-deprecated flags are in that comment block. Please minimally rearrange the flags, so that deprecated & non-deprecated flags are properly grouped.
/kind cleanup /sig network /priority backlog /good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /kind cleanup /sig network /priority backlog /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I can work on this if no one picked it up already.
Go for it
GitHub didn't allow me to assign the following users: zanetworker. Note that only and repo collaborators can be assigned and that issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry. ‍
<!-- Please only use this template for submitting enhancement requests --What would you like to be : a new addon to addon manager "that automatically installs CSI CRDs if CSIDriverRegistry or CSINodeInfo feature gates are true". But this is either broken or this addon manager isn't active in a cluster brough up with: This should work automatically also for such local clusters. Why is this needed: The CSI E2E tests are now expecting that the cluster has the CRDs and fail when that isn't the case. /kind feature (or perhaps a bug, not sure) /cc
/assign
Is this expected to be installed in other clusters? A lot of clusters are not going to be using cluster/* scripts.
<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --pull-kubernetes-e2e-aks-engine-azure-file pull-kubernetes-e2e-azure-file-windows AzureFile CSI Driver End-to-End Tests: Dynamic Provisioning should create a volume on demand with mount options [] [] [Windows] AzureFile CSI Driver End-to-End Tests: Dynamic Provisioning should create multiple PV objects, bind to PVCs and attach all to different pods on the same node [] [] [Windows] AzureFile CSI Driver End-to-End Tests: Dynamic Provisioning should create a volume on demand and mount it as readOnly in a pod [] [] [Windows] If the randomly generated kubetest resource group has capital letters, the share name will contain invalid characters and fail provisioning. links to appreciated links to specific failures in spyglass appreciated <!-- Please see the deflaking doc () for more guidance! --
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
<!-- Please only use this template for submitting enhancement requests --What would you like to be : Currently, during kubelet bootstrapping, while waiting for the node certificate being signed, there is no logs showing what is happening. I would like to add some logging to for easier troubleshooting. Why is this needed: When troubleshooting kubelet bootstrapping process on a node, if there is some issue with certificate signing mechanism, kubelet will wait for its certificate being signed for without a single line of log. Often, the last lines of kubelet log would look like the following, and kubelet will be in an confusing "infinite" wait. And then, some healthchecking monitor for kubelet may kick in and restart kubelet, resulting in a confusing loop. Adding some logging here will greatly help troubleshooting bootstrapping process on the node side. /sig node
The logging would be in the following func: I think we should control the frequency of logging (by remembering the timestamp of last log).
https://k8s- Failed: [] Container Runtime Conformance Test container runtime conformance blackbox test when running a container with a new image should not be able to pull from private registry without secret {E2eNode Suite}
Same with , the fix is under review
https://k8s- Failed: [] Container Runtime Conformance Test container runtime conformance blackbox test when running a container with a new image should not be able to pull from private registry without secret {E2eNode Suite}
https://k8s- Failed: [] Container Runtime Conformance Test container runtime conformance blackbox test when running a container with a new image should not be able to pull from private registry without secret {E2eNode Suite}
The fix has been merged , close this one. Feel free to reopen if this happens again.
https://k8s- Failed: [] Container Runtime Conformance Test container runtime conformance blackbox test when running a container with a new image should not be able to pull non-existing image from {E2eNode Suite}
https://k8s- Failed: [] Container Runtime Conformance Test container runtime conformance blackbox test when running a container with a new image should not be able to pull non-existing image from {E2eNode Suite}
https://k8s- Failed: [] Container Runtime Conformance Test container runtime conformance blackbox test when running a container with a new image should not be able to pull non-existing image from {E2eNode Suite}
The fix has been merged , close this one. Feel free to reopen if this happens again.
<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: Flakes: Full failure: Which test(s) are failing: Since when has it been failing: Since merging Testgrid link: Reason for failure: Serial slow tests are failing with: and the parallel tests are flaking with: Anything else we need to know: I would not expect kubelet to crash if the node if the plugin is not present. /sig windows cc:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
It seems to me that it is not related to a plugin being not present, but that it cannot open the endpoint where potential plugins could register to. In other words, I suspect this error happens independently from any plugins. If we can trust this message , then this would indicate that it tries to open that listening socket twice. This would mean that is called twice. If this is the case, the question would be: Why? It would be interesting to see if shows twice.
There is an issue on some Windows not updated. When you restart kubelet and the socket file So the workarround, is to remove before restart kubeblet
And why tests on the PR didn't trigger this error ?
Question: Is it legal that is called twice? It seems that the function is called from , which is analogous to the Linux implementation Calling twice looks fishy to me. It seems to me that this problem is not caused by, but only exposed by the device plugin change
You mean that kubelet is initialized twice on the same computer, causing this error the second time? How is this handled on Linux? If this is allowed we could simply delete if present before opening it again.
Are you sure it's call twice ? because on windows is not present but replace by you can see the first line comment and .
A simple test is: kubelet and check the socket file is still present kubelet you got the error, stop it socket file kubelet again, normally it's working Maybe the fix can be add in
I think your explanation is very plausible: That it is not called twice within a single kubelet initialization, but kubelet is initialized again for some reason on the same node. (It still means that it is called twice, but I didn't think about the possibility that kubelet is initialized twice.)
There are various ways of fixing it. But I would be interested in learning how this is handled under Linux. To choose the same path for Windows, too.
Thanks for your quick response and analysis :+1: It would be interesting to see if shows twice. You can find the kubelet logs (and many others) for the windows nodes in the "artifacts" tab on This set of tests runs configuration for GMSA tests which as part of the set up of the domain controller, causes a node resets. This explains why one of the nodes seems ok (it isn't randomly selected as part of the domain machines) and why the parallel tests don't cause a full failure (they don't cause node resets). Not all clusters use that script. This set of tests don't use them either (uses aks-engine which has a different start up script). IMHO this is a breaking change scince any cluster that has a node restart would fail to come back online. and kubelet should be able to restart (because of node restart or crash for another reason, etc.) It looks like there is code to clean up old sockets but it is not working on windows: Given there is clean up code there and it's a breaking change I think this this should be fixed in kubelet.
Oh yes, I missed that! (I should have looked only a few lines above the line causing the error.) It at least looks exactly like the code we need. The function is implemented in the same file: But why it is failing to work under Windows? Do you have an idea? Yes, sounds good.
Ok so on windows failed here
So this exemple reproduce the error. the failed only on socket file not on regular file ! Link to
It's bug on windows 1903 not present on windows 1809 The fix can be replace by
Very cool!
Just for interest, how did you find
I hate Windows ;-) and I googled
It's bug on windows 1903 golang/go (comment) not present on windows 1809 This set of tests are running on 2019 LTSC (also known as 1809). If this is only an issue on 1903 is there something else happening?
Sorry, I read to quickly, there is a change of behaviour between 1809 and 1903 about unix socket, If my understanding is good on 1903 unix socket is like regular file but not on 1809, so it's why os.Stat failed on 1809. but I tested on Windows 10 20H2 and the issue still here with
So in summary, the key point is: PR fixes this issue