jiebi/CodeConvo huggingface dataset:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: We created a LoadBalancer service with type 'nlb' with two listeners forwarding traffic on two different ports. The service was created correctly along with the target groups and was functionning correctly. Around 20 hours later, no traffic was flowing from clients to any k8s nodes. While troubleshooting we discovered that the k8s node instance security group was lacking the input rules allowing traffic to the node ports for this LB. We deleted the service and recreated it. The service started working normally again, but we found out that the two target groups were missing their k8s tags. The logs contain the following error: What you expected to happen: We expected the NLB target groups to have k8s tags, so that k8s could remember it created them. How to reproduce it (as minimally and precisely as possible): Create a k8s service with NLB: Add a simple deployment that is matched by this service. Anything else we need to know?: Looking at AWS cloudtrail we could see the following error event: So the API request was called with two Resource ARNs, which is confirmed by the code: Since we have two target groups, there are two resources to tag. Except that the AWS API while allowing to send more than one resource ARNs doesn't allow it returning a validation error. This could be solved by doing one per target groups. Note that this can't happen on a Classic LB because there's no target groups.
I believe that those missing tags on the target groups are the root cause of the deletion of the security group rules allowing traffic to reach the nodeports, with this (hypothetical and unproved) scenario: 8s creates the target groups, the NLB, the listeners, the sg input rules 8s fails to tag the target groups, but doesn't rollback anything a subsequent convergence loops, k8s notices that there are two k8s-managed Security Group input rules, but those don't match any Target Groups or match Target Groups that are not managed by k8s (because they don't have the owner tags) 8s deletes those rules Unfortunately we don't see the rule deletion in the controller manager logs, but it appears in Cloudtrail: Let me know if you have any questions, Environment: Kubernetes version: 1.10.7 Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): CoreOS 1967.3.0 Kernel (e.g. ): Linux 4.14.88-coreos Install tools: home grown Others:
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig aws /assign
Hi, the root cause of your problem should already be solved by It's cherry-picked into old versions and available as v1.11.7, v1.12.5 and v1.13.3(hasn't released yet)
, Thanks, you're most certainly right about the root cause of the issue being . It indeed looks exactly like what we've experienced. Still there's the problem of target groups not being tagged.
When kubelet is running if you create two volume plugin directories here: /usr/libexec/kubernetes/kubelet-plugins/volume/exec /usr/libexec/kubernetes/kubelet-plugins/volume/exec/cifs /usr/libexec/kubernetes/kubelet-plugins/volume/exec/nodeagent~uds You will see these expected logs: Now if you copy over the cifs plugin file to cifs directory and then copy the uds plugin the errors go away. The pods using the uds volume will function but any pods attempting to use the cifs volume will fail with the following error from kubelet: desiredStateOfWorld. err=failed to get Plugin from volumeSpec for volume "pv" err=no volume plugin matched Restarting kubelet fixes this problem as then both plugins are present and properly . A functional volume plugin should be to kubelet even if another volume plugin is missing or invalid. Fixing the invalid volume plugin currently only adds the one that was invalid and fixed and other valid plugins that were previously present are skipped. Start kubelet service on a node. Create two directories for two flex volume plugins in the /usr/libexec/kubernetes/kubelet-plugins/volume/exec/ directory. Eg. nodeagent~uds and cifs Kubelet logs will now start complaining that the plugins are missing. Add the files to the plugin folders and then start a pod using the plugin. Notice that only the last plugin file copied is available to be used by pods. The first plugin will generate this error if used by a volume "err=no volume plugin matched" Performing a touch on the folder of the missing plugin will cause kubelet to properly index it. <details</details<detailsLocal kube cluster </details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig k8s-infra
/sig storage /remove-sig k8s-infra
Not much info on why or how it failed. cc
Cc On Apr 28, 2016 5:08 PM, "Mike Danese" wrote:
cc
cant open the link, would you mind paste the content?
can you access http://ci- ?
It failed due to bind address already in use. The same address were used in example/apiserver (which runs befor federated-apiserver) as well. I wonder if the tests are run in parallel? If it is the case, I can change the port to other values (8082 for example) Otherwise, it may indicate that the address:port was not proper released for some reason
Yes, tests run in parallel. Can you perhaps use a random port number each time? This will reduce the probability of collision to nearly zero (close enough to zero for practical purposes). See . also provides other background reading, but is focussed on end-to-end tests. Much of the discussion applies to unit tests also though.
I think changing the Insecure Port at from 8081 to some other value (say, 8082) should be fine.
Which should I use? 8082 or random? Personally I wouldn't be so optimistic about using random port number since no matter how small the probability is, it will happen eventually, which would be a surprise to those who's not aware of this.
Yes I am fine with changing the port to 8082. Feel free to send me the PR for review.
If you specify port 0 when you bind, the kernel will hand you an available port so there won't be a conflict.
We also want to know the port to be able to curl it in the test.
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: namespace disappears in the output of What you expected to happen: namespace exists How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: It is related to Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig cli
What happened: We enabled feature in our large-scale cluster, and we found client(Kubelet) may got error(500 http code response) like after received GOAWAY frame. I had already re-produce this bug and open an in golang repo. So far I think this bug only occur with a small probability when client request server in high concurrency use a same httpclient instance, and it's a golang sdk bug in http client. Before I solved this issue, I suggest do NOT enable GOAWAY() feature. As it's a golang sdk bug(currently I known and doubt) in http client, revert GOAWAY feature(codes) is not necessary.
cc
/sig api-machinery
Thanks for the report.
Based on not setting the field, I think it defaults that way, so that's good. We will leave the option and filter so that when golang updates its client, we will have a simple way to turn it on and execute it.
Sure. I am trying to fix the golang http client issue and will update progress at . After golang http client issue fixed, I will add more concurrent test cases to see if there are any other bugs may occur by GOAWAY feature. BTW, we still enable GOAWAY feature in our production cluster, because there is no any evidence told us this error will influence SLO, as all most components(kubelet, controller-manager, scheduler) is controller work mode and will re-try after got error. /assign
Golang version or required, which will release at 07/01/2020.
Kubernetes v1.19(or master branch) now is using golang 1.15, and golang 1.15 has fixed the issue. And is on going. Let me add more test cases to ensure the error will not happen again.
I think we should re-open this issue because of . /re-open
On 2k nodes, this test took 53m. We should either do it in batches of nodes, or just subsample the nodes.
And it probably doesn't need to test 4 different variant command sets against all of the nodes, either. :P
When you asked me about the 2k node test, this was the test that was running. I'm not sure why other tests weren't also running in parallel or if this was just the final long pole, but it's a beastly test. PR incoming.
- thanks for looking into it. THe PR is already lgtm-ed by me.
Successfully ran containers with "onFailure" restartPolicy are restarting after purging oldest docker containers. Kubelet purges/removes oldest containers periodically from docker. After purging, we loose the exit code and history of previously ran containers and restart containers as if they were never run.
Thanks & I can help fix this. Let me know. Thanks
Thanks for reporting Just to make sure I understand the issue: You ran some containers with an "onFailure" restartPolicy. When the Kubelet purged old containers the container was restarted (unexpectedly).
Ah, after reading again I think the issue is that after the containers are purged, when the container exits successfully it get restarted although it shouldn't. Is that correct?
After reading through the code I found a bug in the implementation that is most likely causing this error. Should have a PR soon.
Yes. Correct. After initial container run, it should never be run again, since it already ran to completion. Will wait for your PR.
PR is out. This should fix the issue. Feel free to test if you can :)
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --It is not a secret that . In my opinion the main reason for the slowness is the flexibility of the inter-pod affinity/anti-affinity rules. However, in most practical scenarios users do not need all the flexibility. For example, people often do not use arbitrary labels in topology key of the affinity rules. They often use node, zone, or region as the topology. Node in particular is used often. That's why I am proposing a fast path for the case that topology key is Node. Today, the affinity/anti-affinity predicate does not have any distinction for topology keys and in the cluster regardless. The logic should recognize "" and scan only pods on the same node for this topology key. Is this a BUG REPORT or FEATURE REQUEST?: FEATURE REQUEST /kind feature /sig scheduling cc/
ref/
/assign
GitHub didn't allow me to assign the following users: misterikkit. Note that only can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
We found some other simple optimizations that had a significant impact on the affinity predicate. I will upload PRs for those as well.
Reopening this as the performance of anti-affinity still doesn't seem to be satisfactory enough on large clusters (i.e 5k nodes) - and it's causing them to fail to startup. Ref: issue I can help with evaluating any performance improvements if needed.
The change sent improved performance of the predicate, but apparently it is not enough to make our large scale scalability tests happy.
This issue is not about general performance improvment of affinity/anti-affinity. It was to implement a particular improvment which is already implemented. There are other performance improvement issues. I am going to close this one.
Sounds good. Thanks.
How come with kops 1.10.0 I still see all kube-dns pods on the same node because the kube-dns deployment doesn't specify any anti-affinity rule?
I think you are correct. you can double-check with the option to . I have a v1.9 cluster in front of me, and kube-dns has no affinity rules. (I didn't use kops though)
Which jobs are failing: N/A - AFAICT CI jobs works fine Which test(s) are failing: Since when has it been failing: no historical record on CI Testgrid link: N/A Reason for failure: I believe the tests make too many assumptions around the hardware, and the code is actually OK. to be confirmed. Anything else we need to know:
/sig node /sig testing
also affects the recently topology manager e2e tests. My PR uses a different approach to check resource alignment so is not affected. /cc
I'll work on a PR to fix this issue ASAP
Thanks for opening this issue. We knew about this, just had not opened an issue to track it, so thanks for doing that! Deepthi has been looing at CPU Manager tests, not sure if she has some PR in progress or not. I'll make sure to forward this to her (can find her user id here ....) in case she has something in progress
I wasn't able to find automatically create issue for this, but I may not know what to search for. Is our bot creating issues for unit test failures? - this failure looks to me as something in client testing libraries.
If it's exposed in JUnit output, then the bot should be making issues for it.
It's not exposed in junit due to :(
Lots of failures in , probably all this bug.
Yeah this seems to be happening a lot. Raising priority.
I thought was reverted but it's not. I'll revert it.
golang 1.6 doesn't fix the race:
Sent .
hi, we have 3 master nodes , one of nodes errors output this err kubectl -n kube-system logs kube-apiserver-loshu-k8s-master02 k8s version lb proxy haproxy+keeaplive on-premise ubuntu 22.04 kubeadm containerd calico
/sig api-machinery for triage
/cc
Yes, looks like a real bug, an oversight in logic. Will fix.
Will this fix be backported to v1.26?
/cc hello, I want to know what the scenario of this problem is, what if I reproduce it? thanks
I have k8s 1.26 and recently got same error.
/triage accepted
I have observed the same error on v1.27.6
I have observed the same error on v1.28.3
I have observed the same error on v1.28.4
Try to update to the latest patch version.
it seems fixed in See vs
yep, it's been restored, but I want to know the reason it happens
There are not e2e tests for some DNSPolicies see: /help /sig network
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /sig network Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/kind bug What happened: If you start with a minimal Debian image and attempt to install kubernetes- as an artifact from a Bazel build, it will fail with: This has been a known issue in the past, and there are workarounds for CI in place, but it's still unexpected behaviour () What you expected to happen: kubernetes- installs. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): master Cloud provider or hardware configuration: n/a OS (e.g. from /etc/os-release): debian Kernel (e.g. ): n/a Install tools: Others: /sig cluster-lifecycle
hm, this might be a bug in our "fast" tar builder (). if I revert to the bazel-supplied version: then the tarball looks more like we'd expect:
I'll dig a bit more. I started with the assumption that this was a Bazel bug needing a workaround.
cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale /assign This should probably still be fixed.
/assign
/lifecycle active
Once is ok & merged, referenced will need updating in
Oops. Didn't realise this issue was getting spammed by my bazel testing.
yep, better not include references in commits. is this resolved?
Yes, fixed this.
we still need to bump the kubernetes/repo-infra dependency in WORKSPACE though. I'll open a PR to do that for kubernetes/kubernetes.
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: The pod name is missing in suggested cmd usage when executing : What you expected to happen: The suggested cmd usage should be a complete cmd like the one got from How to reproduce it (as minimally and precisely as possible): Use to attach any pod/deployment/service, the PodName will be missing in the suggested cmd usage like above. Anything else we need to know?: It works fine for kubectl < v1.12 Environment: Kubernetes version (use ): v1.13.1 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cli /assign
The docs for the Watch interface are a little ambiguous: "If an error occurs or Stop() is called, this channel will be closed, in which case the watch should be completely cleaned up." That can be read two ways: When the channel is closed, the watch is already completely cleaned up When the channel is closed, the caller must call Stop() to clean up resources. If it's the first, can I suggest we rephrase to "If an error occurs or Stop() is called, this channel will be closed. It is not required to call Stop() to free resources when the channel has been closed." If it's the second, can I suggest we rephrase to "If an error occurs or Stop() is called, this channel will be closed. On observing the closed channel, the caller must call Stop() to free resources." /sig apimachinery
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : The docs for the Watch interface are a little ambiguous: </details
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : The docs for the Watch interface are a little ambiguous: </details
/assign Mike would you be interested in taking a look?
OK, I will take a look.
When we populate the addresses of the nodes in we call NodeAddresses, which for the gce provider gets the external IP from the which as expected puts the IP address of the master into the external IP field. The Legacy IP field is still filled in by asking the cloud provider for the instance data, so it contains the public IP of the node. /cc
If we merge this should get fixed, because the external IPs will be calculated on the nodes rather than on the master.
Since we are going to delay adding until I can rebase on top of it seems like it's worth fixing this separately.
This is blocking so assigning to him.
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Nodelifecycle controller does not set NoExecute taint to NotReady node when kubelet has been stopped What you expected to happen: effect: NoExecute key: Nodelifecycle controller sets NoExecute taint to NotReady node when kubelet has been stopped How to reproduce it (as minimally and precisely as possible): systemctl stop kubelet Anything else we need to know?: I use default args to start kube-controller-manager. I see log from kube-controller-manager: Environment: Kubernetes version (use ): [root ~]# kubectl version Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.6", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-07-15T16:58:53Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.6", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-07-15T16:51:04Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: OS (e.g: ): [root ~]# cat /etc/os-release NAME="CentOS Linux" VERSION="8 (Core)" ID="centos" IDLIKE="rhel fedora" VERSIONID="8" PLATFORMID="platform:el8" PRETTYNAME="CentOS Linux 8 (Core)" ANSICOLOR="0;31" CPENAME="cpe:/o:centos:centos:8" HOMEURL="" BUGREPORTURL="" CENTOSMANTISBTPROJECT="CentOS-8" CENTOSMANTISBTPROJECTVERSION="8" REDHATSUPPORTPRODUCT="centos" REDHATSUPPORTPRODUCTVERSION="8" Kernel (e.g.
): [root ~]# uname -a Linux liabio 4.18.0-147.5.1.el81.x8664 SMP Wed Feb 5 02:00:39 UTC 2020 x8664 x8664 x8664 GNU/Linux Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cluster-lifecycle /wg cluster-lifecycle
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig sig-cluster-lifecycle /wg sig-cluster-lifecycle /committee sig-cluster-lifecycle Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-sig cluster-lifecycle /sig node
Those labels are not set on the issue: <detailsIn response to : /remove-sig cluster-lifecycle /sig node </details
is NoExecute the expected taint for this condition?
- yes it is when running the taint-manager: evictions are usually safe eviction as we don't force delete pods
ref
I see the same, a node which has node condition set to Unknown, does not get an Unreachable taint with NoExecute effect. However I only see this on 1.19, works fine till 1.18.10 Till 1.18.10, such a node gets two Unreachable taints, one with effect NoExecute and one with effect NoSchedule. But on 1.19.3 it only gets the NoSchedule taint. I see that this is the only commit made in that area between 1.18 and 1.19: , could be related?
It totally breaks production readiness of all recent releases, yet doesn't seem to be regarded as a top priority issue. WTF?