jiebi/CodeConvo huggingface dataset:
I may try this on some older kernel like from CentOS 7 Added exactly the same IPVS configuration here: And when running this 'gist/program' pointing to the IPVS from the DNS (fd00:10:96::a port 53 UDP) got the expected result: Maybe trying to run this also in some other scenarios could clarify if this is something related to Netlink API changed between Kernels, something with the used library, etc.
After some study, I find that this issue most likely is caused by the low linux kernel version. I have reproduced this problem on . Accroding to the code, netlink will try to get attribute, but I find the kernel just does not support it :( This is the defined in (kernel 3.10) No attribute is defined! But in new kernel version, the is defined like : Obviously, the kernel has some attributes (IPVSDESTATTRADDRFAMILY, IPVSDESTATTRSTATS64...). That is why kube-proxy works well on systems with a higher version of the kernel. So we can address this issue by upgrading our linux kernel. Not sure which kernel version is the minimum requirement, maybe we should document something about this.
Here is the defined in :
Good finding. So, we can conclude that this really is kernel version dependent. It is hard to know what to do about it. We must check the family. I would propose that we make it work for ipv4 with some fallback and put a requirement on kernel version for ipv6-only/dual-stack. But the decision is others to make. /cc
/cc
The symbol in the kernel tree The for service family seems very old. Since the service must have the same family as the real-servers perhaps it can be used for compatibility with old kernels?
The commit that adds the symbol; Seem to go into linux-3.18
I did not know the minimum required for the kernel version, I was using the stable and official version of RHEL7. My kernel version is: Currently the max version in the repository is: According to the documentation (), I should switch to RHEL8 if I want a recent kernel :-
There is no minimum kernel version requirement afaik. So I think this must be fixed for ipv4-only clusters IMO.
However ipv6-only and dual-stack are in "alpha" so a requirement on kernel version might be ok, but I can't say myself. A complication is that the problem is not in k8s itself but in a 3rd party lib.
ipv6-only is beta since 1.18 :smile:
/assign
I opened a PR to fix this in moby/ipvs If someone w/ a cluster on the older kernel version can test it that would be great, otherwise I will setup a cluster to validate this some time this week.
Instead of bootstraping a whole cluster right now (pretty busy here) I can use the Gist I've mentioned to check in a CentOS 7.1 if it faces the problem before and after the patch. At least to have some more test right now
with a small test, it worked fine. Environment: CentOS 7.1 - Kernel 3.10.0-229.el7.x86_64 + ipvsadm 1.2.1 Pre-loaded some IPv6 rules into ipvs: I've used code, searching for the following structure: I then call ` to get the IPVS object, and then try to fetch the destinations. The following are the results: Before patching the ipvs module: After patching the ipvs module (pinning to your repo/branch) Also I've made those tests with the structure but searching for TCP, and for kube-apiserver (that have only one destination) and it worked fine in all those scenarios. If wanted I can improve the gist to pass the parameters as command arguments instead of hardcoded (sorry, did this in a rush to test)
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry. ‍
/remove-triage unresolved
Thanks just waiting on now
same issue,sloved by update kernel to 5.X
I didn't want to wait, I updated to kernel-lt in the version 4.4 . The problem is solved on my side :)
Thanks for confirming. We'll still try to get in since I'm sure other folks with older kernels will run into this issue.
/retitle parseIP error with IPVS proxy
upgrade your system kernel to 4.xx edition
FYI: just merged, need to run some validation & cut a new release (v1.0.1).
When running CSI migration tests focussing on multi-attach scearnios with GCE PD inline volumes that are read-only, the following error pops up in the logs and the second attachment takes 6 mins: For read-only inline volumes, the above warning/error around is unexpected with CSI migration enabled and needs to be root-caused/resolved. What happened: CSI migration tests for read-only inline GCE PD volumes occasionally fails with the above messages in the logs when executing tests like What you expected to happen: The tests and multi-attach scenarios to execute smoothly with no warnings in the logs. How to reproduce it (as minimally and precisely as possible): Execute with focus set to and run a few times. Anything else we need to know?: This has been observed with CSI migration enabled and the tests exercising read-only inline volume scenarios. Environment: Kubernetes version (use ): v1.16 Cloud provider or hardware configuration: GCE OS (e.g: ): Kernel (e.g. ): Install tools: kube- Network plugin and version (if this is a network-related bug): Others:
/cc
/sig storage
Looks like hard coding access modes to ReadWriteOnce during translation for GCE PD at is causing isMultiAttachForbidden to return leading down to We need to set the access mode to ReadOnlyMany for in-tree inline volumes like in GCE PD (as well as others like EBS that support multi-attach) depending on instead of always setting it to ReadWriteOnce.
I can pick up this fix /assign
What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): kubectl token BUG REPORT Kubernetes version (use ): kubectl version Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-03-28T16:36:33Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-04-03T20:33:27Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"} Environment: Cloud provider or hardware configuration: GKE OS (e.g. from /etc/os-release): macOS Sierra 10.12.4 Kernel (e.g. ): uname -a Darwin jacobsimpson- 16.5.0 Darwin Kernel Version 16.5.0: Fri Mar 3 16:52:33 PST 2017; root:xnu-3789.51.2~3/RELEASEX8664 x8664 Install tools: Others: What happened: I set up a second user, then I use to authenticate as the second user. The parameter is ignored and the authentication token from the file is used instead, so I'm authenticated as the user configured in the config file, not as the user set on the command line parameters. I further verified this by commenting out the section in the config file with the authentication token and then the parameter starts to work as expected. What you expected to happen: I expect the command line parameters, when present, to override the config file settings. How to reproduce it (as minimally and precisely as possible): At this point, the error message should be indicating kubectl tried to authenticate with the account. Something like this: Now edit , search for the section for your cluster and comment out the whole section for the test cluster. Now it should be possible to run these two commands. The one using the user_token will fail with the error message indicating the user name.
Anything else we need to know:
The token arg on the command line overrides the bearer token field in the config file, but I'm not sure the precedence of the token arg and the auth provider in the config file. cc
All auth providers currently override the "Authorization" header even if it's set. Should be an easy-ish check to add. Long term we probably want to have the auth provider stop wrapping the transport and just act like a token source , that way some central code would handle the HTTP headers and we wouldn't have to enforce this on a per plugin basis.
So the tokens from command line should be used, right? if so I can send a pr to fix this.
the client auth plugins shouldn't override the "Authorization" headers. Yeah feel free to send a PR.
Is this a regression or why 1.7?
bug is pretty self contained in the GCP client auth plugin and the fix is short Not strictly a regression, but something that popped up over this cycle as a bug impacting users. Is that reason enough to get it in during the 1.7 freeze?
What happened: A custom CSI driver implementing returns a failure response, indicating that it did not complete its intended steps to prepare the volume for the mounting pod. Let's say that these steps involve a filesystem at the target directory, data from a service, and that data to a set of files in the mounted filesystem. If the first filesystem mounting step succeeds, but either of the subsequent two steps fail, the volume is still not ready for use by the pod. However, if this CSI driver returns a failure response, and the CSI mounter comes back and, before invoking the RPC again, it finds a filesystem already mounted at the target directory, , and ceases further calls to to allow the CSI driver to try again. What you expected to happen: If a CSI driver returns a failure response to a RPC, Kubernetes should honor that declaration of failure and invoke the RPC again, disallowing the mounting pod from starting until this RPC succeeds. Even though it's possible to partially work around the current behavior by having the driver mount the filesystem as late as possible, and then attempting to unmount it if fails to populate the volume with its intended data, doing so makes writing drivers more complicated and forces them to do extra work. How to reproduce it (as minimally and precisely as possible): In a CSI driver (even one that serves just ephemeral, inline volumes), implement to mount a filesystem at the target path, then return a failure response. Confirm that Kubernetes does not invoke again, because there's now a filesystem mounted at the target path. Anything else we need to know?: This topic came up for discussion in the "Kubernetes" Slack team. There, helped investigate this complaint, and to discuss changing this behavior. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: AWS EC2 OS (e.g: ):Kernel (e.g. ):Install tools: kubeadm Others: csi-node-driver-registrar container image tag: v1.0.2
/sig storage
While the plugin should try to clean up as much as possible, I agree that kubelet should not be assuming that if there's a mount point, the publish is successful. NodePublish is supposed to be idempotent so it should theoretically be safe to call it when it's already mounted, although there is the possibility that plugins may have written against the K8s behavior.
For the mount point checking here before calling NodePublishVolume, I am wondering it is just for performance optimization, or it has other purposes. Have concern about some drivers might depend on this check if they are not implemented with idenpotency.
Those drivers would be operating in violation of the specification. I'd prefer that we don't penalize already conforming and yet-to-be written drivers in order to accommodate these existing drivers—if they even exist.
More fallout in last night's testing: I introduced a bug into my driver that caused it to panic during ("Writing that data to a set of files in the mounted filesystem"). My handler function is set up to unmount a successfully mounted volume via if it doesn't complete steps 2 and 3 above, in order to deal with this issue's principal complaint. My driver container would crash due to this panic. The kubelet would call on my driver twice, it would crash twice, but the pod still started running. I'm not yet sure whether or not the filesystem got unmounted successfully via the on the first call, but somehow the pod started despite my handler function never completing successfully. I'm going to try to get some more debugging time on this later today to see if I can figure out what the kubelet was doing there.
Studying my code more, I see that my function only unmounted the mounted directory when the containing handler function was returning an error. When unwinding due to a panic, the named return value of type I was inspecting was still nil. Revising the criteria in that function covers both failure reasons (returning an and unwinding due to a panic). Regardless, this further underscores the problem: If a driver makes it as far as mounting a filesystem at the target directory, but fails afterward—including the process crashing and never responding positively to the RPC—the mounting pod will start. That's likely to then cause the pod to fail, for lack of the files it expected to find mounted.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity.
Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
What happened: When kube-proxy is configured with a cluster-cidr range, it drops packets with an INVALID conntrack state for all sources on the KUBE-FORWARD chain. This is meant to prevent spurious retransmits in long-lived TCP connections to a service IP, but it prevents clusters from having asymmetrical routing on private subnets with services outside of a cluster. What you expected to happen: Packets on the KUBE-FORWARD chain with an INVALID conntrack state should be dropped for only sources inside of the cluster-cidr if set. How to reproduce it (as minimally and precisely as possible): N/A Anything else we need to know?: Change to drop INVALID conntrack packets was in Backing out that commit and compiling is my current workaround. Environment: Kubernetes version (use ): 1.17.2 Cloud provider or hardware configuration: OS (e.g: ):Kernel (e.g. ):
/sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry. ‍
I am reading but I don't fully grok what caused the problem in the first place, or why we think "only drop if it was in-cluster" is a correct fix? Edit, OK I get the original problem, but still don't understand why packets from outside the pod range would not exhibit , thereby requiring yet another revisit of this problem. Is this a conntrack bug? Should it be applying the DNAT?
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry. ‍
I can try to give a bit more context to our setup, but the knowledge I have around this all dates back to Feb, unfortuately, so I may have forgotten details. We were replacing our mesos cluster with a kubernetes cluster in Feb and using a custom ingress solution (HAProxy w/an outside process configuring it for pod ips / mesos ip-ports) for both. The ingress nodes lived in the private subnet, but outside of the kubernetes cluster cidr range. When we upgraded from kubernetes 1.14 to kubernetes 1.15 (the first version with PR ) our ingress could no longer talk to any pod ips.
I discovered that the KUBE-FORWARD chain had a rule to drop invalid packets, which was in Compiling kubernetes 1.15 without this commit fixed our problems, and we have been carrying that patch forward ever since. I would like to get rid of our forked kubernetes, hence this ticket. When I was writing this ticket up, I also found also mentions the same issue with the iptables dropping INVALID conntrack state packets near the end, regarding asymmetrical routing. Edit, OK I get the original problem, but still don't understand why packets from outside the pod range would not exhibit , thereby requiring yet another revisit of this problem. I think you are correct, I can imagine packets from outside of a cluster could exhibit the same behavior as packets inside the cluster and cause spurious resets. However, I am wondering if kube-proxy should be managing packets from outside of the cluster on a node... That seems outside of the scope of kube-proxy.
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry. ‍
I took a stab at implementing what I think is the correct behavior for kube-proxy in .
/assign
I think that another option is to set
I don't understand how that will help me here. The packets with a ctstate INVALID are not generated due to conntrack saturation (), but due to asymmetric routing in our network. See for an example of a similar setup.
are the packets on your asymmetric routing setup marked as invalid if you set ? it should only mark the ones that are out of window per the description
/triage accepted
I'd love to pin this down, but I don't have the means to set up a repro. Is there a way I can set up a way to experiment?
We were bit by this too after revisiting an old service that had been unused since the heady days of pre-, and would also like to see the proposed patch merged (or a configuration option to prevent insertion of the INVALID state drop rule - either suits us). I unfortunately don't have any useful input as to setting up for reproduction. In our case we have a pretty odd anycast set up with a small pool of (public) addresses managed by calico.
However I do seem to have a dirty workaround for small clusters. By setting notrack (ctstate UNTRACKED?) in the raw table for packets in our specific ranges that would otherwise match ctstate INVALID, the packets aren't dropped. Which may be of help to someone else who finds this issue.
We are seeing something very similar on large file downloads and our issue exactly fits this one's subject. These RESET (corrected after are happening intermittently and I am seeing similar behavior mentioned in the blog mentioned below. I am unsure if someone has found a workaround for this issue or not, so yeah my +1 to this issue being looked up by the community.
We're using as a solution for the connection reset we were seeing and might fix this issue.
Let me try to avoid that this thread derails and see if I can get current status: -proxy install rules to drop INVALID cstate connections, this solves the ~REJECT~ RESET problems are some scenarios, with asymmetric routing per example, that this rules may drop unrelated traffic. The request is to remove the rule that drops those INVALID states or to try to restrict it, and that can cause that people start to have the ~REJECT~ RESET problems and drops again. If we have a reproducer we can be more precise in the final solution, but removing one rule without any strong evidence in kube-proxy is very risky As a workaround for the asymetric routing problem, I suggested to set the nfconntracktcpbeliberal on the kernel: I think that will solve the problems, it will be good if someone can test it and provide feedback
No sadly, its my bad, it is not REJECT, it is RESET that's happening and conntrack has nothing to do with it, mostly (I hope). Kubeproxy is doing what is intended but specifically when there are external agents in play, the packet doesn't sends an ACK back to the response of SYN-ACK. Our case, we hit A to initiate a download, A calls B to get the action completed to generate download data for the user finally via their browser. A and B both are on k8s, we saw A starts and a complete cycle goes for sometime until there multiple parallel downloads requested where few gets completed and few gets a RESET.
On primary debugging, it appears that RESET was getting invoked by the pod but pod is just sending a RST-ACK not RST itself. The missing packet is never found in the default route, unless we see that there is a chance where ACK is missed and we found some lines in tcpdump with missing sequence for ACKs. Thats why i doubt my issue is related here. One more weird coincidence is few downloads have spurious transmission.
Thankyou, I will take a look, but mostly I would like to wait before using it, debug more to understand the actual cause and then act on it.
sorry , I wanted to say RESET, thanks for catching it. If you think is different and you can provide information and how to reproduce it, I suggest you to open a new issue
I will sure do, when it is different. For now I would like to watch and submit my findings if it is related. PS: sysctl -w net.netfilter.nfconntracktcpbeliberal=1 is already implemented and we are testing if it does something helpful. so far it doesn't looks like it. thanks for your help.
/area kube-proxy
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle rotten
I have the feeling we are also affected by this with a standard calico installation: every node announces a /122 IPv6 network to the upstream router the upstream router uses ECMP to route the packet back packets that are sent from* the pod and returned via asymmetric path from outside seem to be dropped
and if you set ?
Same behaviour with set on all nodes, the outgoing pod traffic does not return. And I instantly see packets going back into the drop rule:
:thinking:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.
This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hi, are you still working on this? Same problem happened in our clusters, just because of asymmetrical routing.
/reopen
Reopened this issue. <detailsIn response to : /reopen </details
Also the command suggested by is helping us, but I still think this should be a bug of kube-proxy.
The problem here is: what happens if the solutions are exclusive or we regress? The addition of this rule fixed a legit bug that was documented and there is also a nice blog explaining the details Removing this rule without a repro and understanding what is the root cause of this issue is not really the best way to proceed. I think that people affected should try to find the root cause and explain the problem or just provide a way to reproduce it, so we can fix it
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at .