Change:
The ipvs mode limit the scheduler algorithm to : now in new kernel have this new algorithm : the are specialy interesting... that be in kernel 4.18...
/sig network /area ipvs
/assign
This seems non-controversial on its face, but a) how do we know whether the kernel supports it or not; and b) if every node makes independent decisions, doesn't this basically become random? /assign uablrek
a) It should be the user's responsibility to ensure that the configured ipvs sheduler is available on all nodes This sounds rude, but please read for some rationale (and ). Exception may be the which is the default, but an installation with but without the sheduler is extremely unlikely. If that relaxing of the sheduler pre-test is acceptable, I would say that any current and future ipvs sheduler shall be allowed. Just treat them as a string. But it should be documented in the release-notes and some place else, but I don't know where.
b) For all shedulers that make decisions based on local state, yes. (that includes btw) Well, not really random ... For instance if is used and a new endpoint is addred, then all ipvs instances will get an "empty" target and will select it for new connections and may overload the newcomer, or it will become "statistically correct" over time. But IMHO users should be allowed to try. Some sheduler may be very useful in their environment. That will (already has) of course spawn some issues when it doesnt work as expected (for the reason in b) for instance), but allowing all shedulers will not change that.
So, my proposal is: Drop the scheduler module check Treat the configured scheduler as a string in the ipvs proxier Document
The module check can be replaced with a function check, e.g. insert/delete an ipvs dummy rule.
people is trying to reduce privileges on containers these days, does it requires more privileges than today or is the same?
The same. Kube-proxy configures ipvs already. This would only add an extra probing insert/delete on start. Even on first start (if ipvs is empty) with some care.
Could even be less privileges if "modprobe" requires any privileges (now or in the future)
What is your use-case? I have wanted to check the sceduler in ipvs, but havn't got time yet. The load-balancer is stateless, but is not (well, at least not normally). A problem Maglev tries to solve is if the number of Application Servers (AS) (endpoints in K8s) changes, the updated hash buckets must be as few as possible. But if the load-balancer is stateful and stores the connections (like ipvs or the Linux conntrack used in proxy-mode=iptables), no existing connections are affected, no matter what hash algorithm is used for new connections. So, for both proxy-mode ipvs and iptables existing connection are not affected by adding or removing endpoints. But... if external traffic is routed to different nodes, e.g. a GW rehashes ECMP targets after a cluster scaling, then traffic is affected. This is easily tested: Setup a service of type LoadBalancer Route external traffic to the loadBalancerIP to node A Setup a number of TCP connections Re-route external traffic to the loadBalancerIP to node B All connections are lost (I have tested). But if the load-balancer would have been a stateless Maglev no connections would be lost. Google Maglev stores connections but only as a cache and getting the best of stateful, existing connections never lost on AS scaling, and stateless, the load-balancing tier can be scaled with minimum impact. This can be the case in ipvs , but is it?
Lars: proposal SGTM. IPVS impl is pretty much your call to make :) On Thu, Dec 22, 2022, 1:37 AM Lars Ekman
I have started working on a PR to remove the module checks. You are the only active reviewer and I will try my best to divide it into small reviewable commits as you wrote in :smile:
PR to remove the module checks I also intend to remove the constraint on ipvs schedulers, which will fix this issue. If you can create a VS with your configured scheduler you can use it, whatever that scheduler may be.
Looks like it is. Please read I have tested "mh" in kube-proxy (with a patch) and first problem is that is hashes on address only by default, so a traffic generator using the same source all the time will end up on the same endpoint. This can be configured but will require special treatment in kube-proxy.
Should we close this now that is in?
/triage accepted
No, just removes the module ckeck. The validator still restricts the usable schedulers
to add afinity without using states. is the finality of maglev to avoid perf on managing state... and is more robust on target lost than other consistant hashing method... is normal is what you whant when you use maglev... to have afinity on ip src... but without need of storing states... is afinity by consistant hashing... cilium use this algorithm too....
the google research paper on the algorythm:
If you can build and use an own please try PR I have tested and "mh" can be configured, and with many src-addresses load-balancing works. However if external ingress traffic is re-routed from one K8s node to another, all connections are lost, same as "rr".
with ipvs the node who receive the traffic is considered a director, if you LB to other node... this node must have default route to the director because of nat masquerading of ipvs... when you use MH/RR with Direct Server Return avoid that problem (no more nat masquerading)... the node respond to the client directly (no need to return back to the first node who receive the traffic). but you need to tunnel the traffic from node who receive the request to the node you forward the traffic (cloud flare use fou protocol for that)... i dont know if kubeproxy already support that sort of configuration... cilium for LB DSR + Maglev use tcp option field to store the Virtual IP that receive the traffic in the packet and nat the destination with the pod ip before sending to the node who host the node (that work in full routing mode), and the node respond to the client directly in replacing thaire ip with the Virtual IP from tcp option field.... in that way no state needed... no tunneling... no complex network configuration.... some has been done on kube-router project: and i think you must use in conjonction with MH/RR to use Direct Server Return if i understand whell... and good exemple of the use case : that use ipvs (use ipip tunnel) to do that....
Regarding "mh" algorithm, I submitted PR a few weeks ago which adds the new load distribution method to kube-proxy. As this commit supports 'mh" with hashing on srcIP and srcPort by default, connections from the same srcIP with different srcPorts will be distributed equally to all EndPoints. If users want to have an affinity on srcIP, they can use "mh" with "Session Affinity" option enabled. The PR is targeted for the main use-cases below: Connection failover from a worker node to another. Scale-in/out worker nodes Rerouting from GWs or external LoadBalancer changes receiving worker nodes of existing traffic flows. I would need to rebase the commit after your fix for this issue PR . PR is now waiting triage state, please verify if it is reasonable to add this new algorithm to kube-proxy.
No, and it probably never will. The proper (only) way to get this into is to write a (KEP). To :
I would also like to see a demo or e2e test working with this setup, it doesn't seem obvious from the comments that this will work out of the box ... The project Kubernetes, as Lars correctly mentions , use KEPs to discuss and document feature, I agree with him and I think this will require to write a KEP
What I meant was your use-case in K8s. Having worked most of 2021 with the Maglev hash algorithm when developing the I know what it can do, but I also know K8s and I think I can say right now; To allow the "mh" in kube-proxy=ipvs will not give you what you hope for.
And a side-remark on DSR. Cilium can do this because they modify packets with eBPF. The traditional way of doing DSR is to actually assign the Virtual IP address (VIP) to an interface in the server (the POD in the K8s case). This must have support in the CNI-plugin, which I guess is the case for as it handles load-balancing and is a CNI-plugin. To add generic support for DSR in K8s is not simple.
I'm new to this community. Thanks for your suggestion about the process of adding a new feature to kube-proxy. I'll prepare a KEP for the "mh" load distribution.
But in any case I would like PR to be merged. For sanity reasons if not others. The for scheduler=mh can/should be in a separate issue+PR. This will allow users to test "mh" in kube-proxy, because; Nobody will take my word for it :smile:
I don't want to sound discouraging but IMO you should not spend too much time on a KEP before presenting it to get some early feedback. I can't see how "mh" can provide the statelessness unless DSR is used, and that may be very hard to get into the . So, I suggest you describe a use-case in the KEP, and what it takes to support it, and present it on a bi-weekly sig/network meeting. And don't worry, people are nice there :smile:
Perhaps DSR is not needed to use with "mh" since I cannot really see any problem with nat masquerading of ipvs if subsequence packets are sent to the second node. With net/ipv4/vs/sloppy_* enabled, ipvs on the second node accepts the packets as an existing connection and then keeps forwarding them to the original endpoint thanks to the mh algorithm. IPVS of course performs snat and "remember" the client IP of the packets as if they are sent to the first node. When receiving replies from the endpoint, ipvs on the second node will be able to forward the replies to the client with the cached client IP in its NAT table. Please correct me if I'm missing something in this regard.
So even if the kube-proxy forwards packets to the same POD (assuming the same mh configuration in node A and B) the receiving POD will reject with a TCP-RST packet.
Since you have written PR you should be able to test this with you own-built kube-proxy. For me all connections are teminated when re-directing ingress traffic from one K8s node to another.
Thanks for your clarification. Now I could see this problem is on the pod side, not ipvs. It seems that having DSR does not help to solve the problem. Though, I suppose this issue will not happen with the following scenarios: UDP traffic Traffic is sent to Cluster-IP instead of NodeIP
With DSR the src=VIP all the way, so it should be OK, even if masq (DNAT) is used. UDP same thing, unless your UDP server has no session and doesn't care about addresses, just the server port. This is however not the usual case. Traffic sent to the Cluster-IP will be have src=pod, yes, but it will always be hadled by the local kube-proxy, so the use-case to move traffic to another node doesn't exist for clusterIP.
Me neither :) Agree - you will have the best experience if do this iteratively. Start with an email to the SIG list. Then a doc discussing the peoblem and high-level solution details. THEN a KEP makes sense.
cilium assign vip also but more higher in the kernel stack with ebpf... in that way is more simple to implemente than in ipvs... juste simple packet manipulation (plus the fact that is working only in full routed mode)... ipvs can do that, helped with some config on each node (configuring tunnel for dsr)... for mh usage without dsr we need to add default route on other node to the actual dedicated kubeproxy as l4 director (one node can be designated as director and move to another node when the first one die). is generaly more for loadbalancer/external ip usage....
Is this an acceptable way? merged. This will allow "mh" to be configured as a ipvs scheduler, but only with L3 hashing (no ports). a new issue like "Ipvs scheduler=mh only hashes on addresses, not ports", and implement it in a new (small) PR with FlagSourceHash as have already done in this issue as "done" since "mh" can be configured and as it's implemented in another way. the work towards a KEP to make "mh" work as you want it to work. Including a K8s use-case, DSR (if needed in some form), setting of sysctl's, etc. I leave item 4. to you and . Item 1-3 will enable "mh" but because of NAT, connections are still lost when traffic is re-directed from one node to another (this can be tested without too much effort. I have done it).
Agree with the approach. fixes the raised issue that it enables all distribution methods supported by ipvs in kube-proxy (including "mh" one). A separate issue for enabling source port hash when using 'mh' sounds reasonable. This is not only applicable for "mh", but also for the "sh" method which kube-proxy is currently supporting as stated in k8s documents. Agree. Sure. I'll need to verify the issue further and check if any easy way possible to avoid the NAT issue on different nodes. IMHO, replacing VIP (virtual IP) with NodeIP seems to be the first option to overcome the issue, but we need to verify its feasibility and impact in k8s.
To allow the "mh" in kube-proxy=ipvs will not give you what you hope for. Nobody will take my word for it :smile: Ok, ok, please let me rephrase: To just allow "mh" in the current ipvs proxier will not give you what you hope for. So, is not enough.
/kind feature

Discussion:
LocalModeInterfaceNamePrefix LocalMode = "InterfaceNamePrefix" ) <del> // IPVSSchedulerMethod is the algorithm for allocating TCP connections and // UDP datagrams to real servers. Scheduling algorithms are implemented as kernel modules. // Ten are shipped with the Linux Virtual Server. type IPVSSchedulerMethod string const ( // RoundRobin distributes jobs equally amongst the available real servers. RoundRobin IPVSSchedulerMethod = "rr" // WeightedRoundRobin assigns jobs to real servers proportionally to their real servers' weight. // Servers with higher weights receive new jobs first and get more jobs than servers with lower weights. // Servers with equal weights get an equal distribution of new jobs. WeightedRoundRobin IPVSSchedulerMethod = "wrr" // LeastConnection assigns more jobs to real servers with fewer active jobs. LeastConnection IPVSSchedulerMethod = "lc" // WeightedLeastConnection assigns more jobs to servers with fewer jobs and // relative to the real servers' weight(Ci/Wi). WeightedLeastConnection IPVSSchedulerMethod = "wlc" // LocalityBasedLeastConnection assigns jobs destined for the same IP address to the same server if // the server is not overloaded and available; otherwise assigns jobs to servers with fewer jobs, // and keep it for future assignment. LocalityBasedLeastConnection IPVSSchedulerMethod = "lblc" // LocalityBasedLeastConnectionWithReplication with Replication assigns jobs destined for the same IP address to the // least-connection node in the server set for the IP address. If all the node in the server set are overloaded, // it picks up a node with fewer jobs in the cluster and adds it to the sever set for the target. // If the server set has not been modified for the specified time, the most loaded node is removed from the server set, // in order to avoid high degree of replication. LocalityBasedLeastConnectionWithReplication IPVSSchedulerMethod = "lblcr" // SourceHashing assigns jobs to servers through looking up a statically assigned hash table // by their source IP addresses. SourceHashing IPVSSchedulerMethod = "sh" // DestinationHashing assigns jobs to servers through looking up a statically assigned hash table // by their destination IP addresses. DestinationHashing IPVSSchedulerMethod = "dh" // ShortestExpectedDelay assigns an incoming job to the server with the shortest expected delay. // The expected delay that the job will experience is (Ci + 1) / Ui if sent to the ith server, in which // Ci is the number of jobs on the ith server and Ui is the fixed service rate (weight) of the ith server. ShortestExpectedDelay IPVSSchedulerMethod = "sed" // NeverQueue assigns an incoming job to an idle server if there is, instead of waiting for a fast one; // if all the servers are busy, it adopts the ShortestExpectedDelay policy to assign the job. NeverQueue IPVSSchedulerMethod = "nq" ) </del> func (m *ProxyMode) Set(s string) error { *m = ProxyMode(s) return nil

} allErrs = append(allErrs, validateIPVSTimeout(config, fldPath)...) <del> allErrs = append(allErrs, validateIPVSSchedulerMethod(kubeproxyconfig.IPVSSchedulerMethod(config.Scheduler), fldPath.Child("Scheduler"))...) </del> allErrs = append(allErrs, validateIPVSExcludeCIDRs(config.ExcludeCIDRs, fldPath.Child("ExcludeCidrs"))...) return allErrs

return allErrs } <del> func validateIPVSSchedulerMethod(scheduler kubeproxyconfig.IPVSSchedulerMethod, fldPath *field.Path) field.ErrorList { supportedMethod := []kubeproxyconfig.IPVSSchedulerMethod{ kubeproxyconfig.RoundRobin, kubeproxyconfig.WeightedRoundRobin, kubeproxyconfig.LeastConnection, kubeproxyconfig.WeightedLeastConnection, kubeproxyconfig.LocalityBasedLeastConnection, kubeproxyconfig.LocalityBasedLeastConnectionWithReplication, kubeproxyconfig.SourceHashing, kubeproxyconfig.DestinationHashing, kubeproxyconfig.ShortestExpectedDelay, kubeproxyconfig.NeverQueue, "", } allErrs := field.ErrorList{} var found bool for i := range supportedMethod { if scheduler == supportedMethod[i] { found = true break } } // Not found if !found { errMsg := fmt.Sprintf("must be in %v, blank means the default algorithm method (currently rr)", supportedMethod) allErrs = append(allErrs, field.Invalid(fldPath.Child("Scheduler"), string(scheduler), errMsg)) } return allErrs } </del> func validateKubeProxyNodePortAddress(nodePortAddresses []string, fldPath *field.Path) field.ErrorList { allErrs := field.ErrorList{}

} } <del> func TestValidateIPVSSchedulerMethod(t *testing.T) { newPath := field.NewPath("KubeProxyConfiguration") successCases := []kubeproxyconfig.IPVSSchedulerMethod{ kubeproxyconfig.RoundRobin, kubeproxyconfig.WeightedRoundRobin, kubeproxyconfig.LeastConnection, kubeproxyconfig.WeightedLeastConnection, kubeproxyconfig.LocalityBasedLeastConnection, kubeproxyconfig.LocalityBasedLeastConnectionWithReplication, kubeproxyconfig.SourceHashing, kubeproxyconfig.DestinationHashing, kubeproxyconfig.ShortestExpectedDelay, kubeproxyconfig.NeverQueue, "", } for _, successCase := range successCases { if errs := validateIPVSSchedulerMethod(successCase, newPath.Child("Scheduler")); len(errs) != 0 { t.Errorf("expected success: %v", errs) } } errorCases := map[string]struct { mode         kubeproxyconfig.IPVSSchedulerMethod expectedErrs field.ErrorList }{ "non-existent scheduler method": { mode:         kubeproxyconfig.IPVSSchedulerMethod("non-existing"), expectedErrs: field.ErrorList{field.Invalid(newPath.Child("ProxyMode.Scheduler"), "non-existing", "must be in [rr wrr lc wlc lblc lblcr sh dh sed nq ], blank means the default algorithm method (currently rr)")}, }, } for _, errorCase := range errorCases { errs := validateIPVSSchedulerMethod(errorCase.mode, newPath.Child("ProxyMode")) if len(errorCase.expectedErrs) != len(errs) { t.Fatalf("Expected %d errors, got %d errors: %v", len(errorCase.expectedErrs), len(errs), errs) } for i, err := range errs { if err.Error() != errorCase.expectedErrs[i].Error() { t.Fatalf("Expected error: %s, got %s", errorCase.expectedErrs[i], err.Error()) } } } } </del> func TestValidateKubeProxyNodePortAddress(t *testing.T) { newPath := field.NewPath("KubeProxyConfiguration")