Change:
<Reopened this valid and in-demand bug after the auto-bot expired it/kind bug /sig network /area kube-proxy What happened: Building a fully static Linux kernel with kubeproxy's desired IPVS (Virtual Server) modules linked statically: CONFIGIPVS=y CONFIGIPVSRR=y CONFIGIPVSWRR=y CONFIGIPVSSH=y CONFIGNFCONNTRACKIPV4=y Causes kubeproxy to believe that these features don't exist because it can't find modules for them: What you expected to happen: If Kubeproxy's desired kernel features are linked statically in the kernel, kubeproxy should carry on and not attempt and throw error messages about modules not being found. Under the hood, kubeproxy should simply check for the existence of the desired kernel features in the /proc tree (ie: , and others) before attempting to look for and load modules. If, after module loads fails, then throw the error messages and switch to iptables. How to reproduce it (as minimally and precisely as possible): and extract the current stable kernel source version (today it's 4.18.9) from your existing running into the source tree as and answer any questions to switch to a fully static kernel and ensure the following exist, and are not : and deselect (box is empty) your boot loader's config () or () depending on your distribution. and select your new kernel with the kubeproxy modules linked statically. kubeproxy and watch it throw the above errors about not finding the modules. Anything else we need to know?: Fully static kernels will not generate so inspecting that file will fail - as well in the kubeproxy source. The gold-standard is to simply check for the features in the active tree first, and consider those as "built-in". The next best options is checking for statements, which tells you if they're built-in. Note that files are almost universally installed by OS maintainers. And finally, you could try checking , which is a compressed version of the externally held .config text file, in a similar manner noted above, however it's up to the maintainer to build that feature into the kernel (so it might not exist). For reference, here's the list of modules kubeproxy wants: Here we see kube_proxy's errant module-centric code that needs to be made compatible w/ static kernels: Environment: Kubernetes version (use ): 1.11.3 Cloud provider or hardware configuration: On premise OS (e.g. from /etc/os-release): Leap 15.0 Kernel (e.g. ): 4.18.9 statically linked Install tools: gcc 8.1 Others: <reopened as it appears still unfixed and in-demandSee how far the team got on it before it was auto-closed here:
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry. ‍
Hi is the issue just the log messages, or do the kube-proxy features actually fail?
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry. ‍
/unassign
/remove-triage unresolved
To check kernel-modules instead of functions is not limited to in k8s. I build my own kernel which was static from the start (with ipvs support). It was a very tiresome trial-and-error time to find out which functions k8s is checking by it's modules. But those who build the Linux kernel probably can probably accept less "help", so a flag to the apropriate k8s programs may be a solution.
I've checked the code, this is only a warning. Maybe while loading the modules, this function could be reduced to something like "check if this exists exists in the kernel builtin modules and append it into the mod list, otherwise try to modprobe it". I've made a PR to it, let's see if this solves the "problem"
/assign
Those are not options, as compilling the kernel doesn' t imply that one will copy the config file into /boot after recompiling it. As an example, the step ` is optional. So the only way right now I've found to be plausible is to verify the existence of the related but what happens if this path changes in the future?
take a look at to see if this is some way to follow Don't think adding a new flag is a good idea (but I might be wrong either, and TBH I've tried ;) )
rikatz, that looks like a reasonable approach. Although the kernel API isn't standardized, the developers do treat /proc as close to stable as possible, and only change existing files and structures as a last resort (they realize that change here ripple across critial userspace, including stuff like systemd). So I think checking proc for the desired features is the first and most reliable approach and then falling back to those in the order that you've suggested in your write up. The error message can even indicate the name of the /boot/config-* file, to help those building their own kernels who might have forgot to provide a correspondingly-named config.
/reopen The patch was a partial solution, We still need to verify if the module loader will support kernels with all builtin
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale Em ter, 14 de jan de 2020 às 07:32, fejta-bot <:

Discussion:
} ipvsModules := utilipvs.GetRequiredIPVSModules(kernelVersion) <del> builtinModsFilePath := fmt.Sprintf("/lib/modules/%s/modules.builtin", kernelVersionStr) b, err := ioutil.ReadFile(builtinModsFilePath) if err != nil { klog.Warningf("Failed to read file %s with error %v. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules", builtinModsFilePath, err) } </del> var bmods []string <del> for _, module := range ipvsModules { if match, _ := regexp.Match(module+".ko", b); match { bmods = append(bmods, module) } } // Try to load IPVS required kernel modules using modprobe first for _, kmod := range ipvsModules { err := handle.executor.Command("modprobe", "--", kmod).Run() if err != nil { klog.Warningf("Failed to load kernel module %v with modprobe. "+ "You can ignore this message when kube-proxy is running inside container without mounting /lib/modules", kmod) } } // Find out loaded kernel modules </del> <ins> // Find out loaded kernel modules. If this is a full static kernel it will thrown an error </ins> modulesFile, err := os.Open("/proc/modules") if err != nil { <ins> klog.Warningf("Failed to read file /proc/modules with error %v. Kube-proxy requires loadable modules support enabled in the kernel", err) </ins> return nil, err }

return nil, fmt.Errorf("failed to find loaded kernel modules: %v", err) } <ins> builtinModsFilePath := fmt.Sprintf("/lib/modules/%s/modules.builtin", kernelVersionStr) b, err := ioutil.ReadFile(builtinModsFilePath) if err != nil { klog.Warningf("Failed to read file %s with error %v. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules", builtinModsFilePath, err) } for _, module := range ipvsModules { if match, _ := regexp.Match(module+".ko", b); match { bmods = append(bmods, module) } else { // Try to load the required IPVS kernel modules if not built in err := handle.executor.Command("modprobe", "--", module).Run() if err != nil { klog.Warningf("Failed to load kernel module %v with modprobe. "+ "You can ignore this message when kube-proxy is running inside container without mounting /lib/modules", module) } } } </ins> return append(mods, bmods...), nil }