Change:
! ref:
/sig node /kind cleanup If any flag found which was removed in , Kindly remove them too.
/triage accepted
/language en
For anyone facing this issue and Kubernetes is not starting, I removed the flag from file and restarted kubelet service. ().
/priority important-longterm
I have run into the same problem and removing it from does not work. Basically this breaks the system
Hi Seems like you are using . If not ,kindly share kubelet version and log. Thanks !!
The version i used was 1.24.0 from the debian repo, downgrading to 1.23.6 solved my issue and got everything running again Let me know if more information is needed?
How you have bootstrapped the cluster? Using kubeadm or different tools
I was using 1.23 and I came up with this issue upon performing updates to the nodes of the cluster (apt-get upgrade). Then, 1.24 was installed and the issue came up and I managed to resolve it as I mentioned above as the logs of the kubelet service were indicating error with the --network-plugin flag.
Behaviour is expected. since remove in v1.24 Please share and the output of ?
: Logs:
Folks, this is an accepted issue. We should update that page to remove mentions of the command line argument. Pull requests are welcome - see for tips on how to contribute.
my cluster is down after doing an apt upgrade. From what I can understand it looks like I have this issue, running version 1.24.0 as well. Is here any instructions on how to downgrade my cluster so that I can get it up and running again? Update: I’ve managed to get my cluster up and running again. Whilst I agree that this comment is off-topic for the exact problem at hand, this may help someone with the problem that I had. The problem is that my K8S install was running on docker, and now dockershim has been removed, which is what stopped my cluster working. I used a couple of posts to get the problem I was having solved. Here’s one of the link (I can’t remember the other) On the workers, I streamlined the changes to this…. The first line should prevent the problem that I had - auto upgrading k8s during an apt upgrade. Updating to 1.24 had the breaking change that caused my cluster to become unavailable. sudo apt-mark hold kubelet kubeadm kubectl sudo systemctl enable --now kubelet sudo su - mkdir -p /etc/containerd containerd config exit sudo nano /var/lib/kubelet/kubeadm- Remove --network-plugin=cni Add --container-runtime=remote --container-runtime- Save Sudo reboot At this point the node should come up. Not 100% sure if this will solve the problem on the control plain, I got that working before streamlining the commands, anything else should be able to be taken from the link I posted above.
I got mine up by simply downgrading the kubelet to latest 1.23.x
I've marked some off-topic comments as hidden. They're not deleted (you can click Show comment to reveal them). Please, folks, do focus on the topic here: fixing the Kubernetes documentation about setting up cluster networking
If anyone has questions about how to propose a fix for , you're very welcome to ask. You can reply here or, if you use the Kubernetes workspace for Slack, ask there in the channel.
I've created a SO page to redirect all the "how to fix my cluster now" questions there: . Should I update the OP to reflect this link and state that this is not a request to fix the cluster's setup?
Duplicated by issue
We should edit so that it is no longer incorrect, and then close this issue.
Anyone who wants to edit that page is welcome to.
/assign
is reviewing PR
remove " --network-plugin"flag work fine.
This resulted in my case just in another error:
same error for me.
same here EDIT: fixed it by "migrating" from docker to containerd by simply: now connect kubelet to containerd directly by editing /etc/systemd/system/kubelet.service.d/10- add after "ExecStart=/usr/bin/kubelet" which looks like: for me. finally restart kubelet.service:
As this bug is resolved, I'm going to lock it to ensure that the record remains on-topic.

Discussion:
Kubernetes {{< skew currentVersion >}} supports [Container Network Interface](https://github.com/containernetworking/cni) (CNI) plugins for cluster networking. You must use a CNI plugin that is compatible with your cluster and that suits your needs. Different plugins are available (both open- and closed- source) in the wider Kubernetes ecosystem. <ins> A CNI plugin is required to implement the [Kubernetes network model](/docs/concepts/services-networking/#the-kubernetes-network-model). </ins> You must use a CNI plugin that is compatible with the [v0.4.0](https://github.com/containernetworking/cni/blob/spec-v0.4.0/SPEC.md) or later releases of the CNI specification. The Kubernetes project recommends using a plugin that is

## Installation <del> A CNI plugin is required to implement the [Kubernetes network model](/docs/concepts/services-networking/#the-kubernetes-network-model). The CRI manages its own CNI plugins. There are two Kubelet command line parameters to keep in mind when using plugins: </del> <ins> A Container Runtime, in the networking context, is a daemon on a node configured to provide CRI Services for kubelet. In particular, the Container Runtime must be configured to load the CNI plugins required to implement the Kubernetes network model. </ins> <del> * `cni-bin-dir`: Kubelet probes this directory for plugins on startup * `network-plugin`: The network plugin to use from `cni-bin-dir`.  It must match the name reported by a plugin probed from the plugin directory.  For CNI plugins, this is `cni`. </del> <ins> {{< note >}} Prior to Kubernetes 1.24, the CNI plugins could also be managed by the Kubelet using the `cni-bin-dir` and `network-plugin` command-line parameters. These command line parameters were removed in Kubernetes 1.24, with management of the CNI no longer in scope for Kubelet. </ins> <del> ## Network Plugin Requirements </del> <ins> See [Troubleshooting CNI plugin-related errors](/docs/tasks/administer-cluster/migrating-from-dockershim/troubleshooting-cni-plugin-related-errors/) if you are facing issues following the removal of dockershim. {{< /note >}} </ins> <del> Besides providing the [`NetworkPlugin` interface](https://github.com/kubernetes/kubernetes/tree/{{< param "fullversion" >}}/pkg/kubelet/dockershim/network/plugins.go) to configure and clean up pod networking, the plugin may also need specific support for kube-proxy.  The iptables proxy obviously depends on iptables, and the plugin may need to ensure that container traffic is made available to iptables.  For example, if the plugin connects containers to a Linux bridge, the plugin must set the `net/bridge/bridge-nf-call-iptables` sysctl to `1` to ensure that the iptables proxy functions correctly.  If the plugin does not use a Linux bridge (but instead something like Open vSwitch or some other mechanism) it should ensure container traffic is appropriately routed for the proxy. </del> <ins> For specific information about how a Container Runtime manages the CNI plugins, see the documentation for that Container Runtime, for example: - [containerd](https://github.com/containerd/containerd/blob/main/script/setup/install-cni) - [CRI-O](https://github.com/cri-o/cri-o/blob/main/contrib/cni/README.md) </ins> <del> By default if no kubelet network plugin is specified, the `noop` plugin is used, which sets `net/bridge/bridge-nf-call-iptables=1` to ensure simple configurations (like Docker with a bridge) work correctly with the iptables proxy. </del> <ins> For specific information about how to install and manage a CNI plugin, see the documentation for that plugin or [networking provider](/docs/concepts/cluster-administration/networking/#how-to-implement-the-kubernetes-networking-model). </ins> <del> ### CNI </del> <ins> ## Network Plugin Requirements The plugin may also need specific support for kube-proxy.  The iptables proxy obviously depends on iptables, and the plugin may need to ensure that container traffic is made available to iptables.  For example, if the plugin connects containers to a Linux bridge, the plugin must set the `net/bridge/bridge-nf-call-iptables` sysctl to `1` to ensure that the iptables proxy functions correctly.  If the plugin does not use a Linux bridge (but instead something like Open vSwitch or some other mechanism) it should ensure container traffic is appropriately routed for the proxy. </ins> <del> The CNI plugin is selected by passing Kubelet the `--network-plugin=cni` command-line option.  Kubelet reads a file from `--cni-conf-dir` (default `/etc/cni/net.d`) and uses the CNI configuration from that file to set up each pod's network.  The CNI configuration file must match the [CNI specification](https://github.com/containernetworking/cni/blob/master/SPEC.md#network-configuration), and any required CNI plugins referenced by the configuration must be present in `--cni-bin-dir` (default `/opt/cni/bin`). </del> <ins> By default if no kubelet network plugin is specified, the `noop` plugin is used, which sets `net/bridge/bridge-nf-call-iptables=1` to ensure simple configurations (like Docker with a bridge) work correctly with the iptables proxy. </ins> <del> If there are multiple CNI configuration files in the directory, the kubelet uses the configuration file that comes first by name in lexicographic order. </del> <ins> ### Loopback CNI </ins> <del> In addition to the CNI plugin specified by the configuration file, Kubernetes requires the standard CNI [`lo`](https://github.com/containernetworking/plugins/blob/master/plugins/main/loopback/loopback.go) plugin, at minimum version 0.2.0 </del> <ins> In addition to the CNI plugin installed on the nodes, Kubernetes requires the standard CNI [`lo`](https://github.com/containernetworking/plugins/blob/master/plugins/main/loopback/loopback.go) plugin, at minimum version 0.2.0 </ins> <del> #### Support hostPort </del> <ins> ### Support hostPort </ins> The CNI networking plugin supports `hostPort`. You can use the official [portmap](https://github.com/containernetworking/plugins/tree/master/plugins/meta/portmap) plugin offered by the CNI plugin team or use your own plugin with portMapping functionality.

} ``` <del> #### Support traffic shaping </del> <ins> ### Support traffic shaping </ins> **Experimental Feature**

## Usage Summary <del> * `--network-plugin=cni` specifies that we use the `cni` network plugin with actual CNI plugin binaries located in `--cni-bin-dir` (default `/opt/cni/bin`) and CNI plugin configuration located in `--cni-conf-dir` (default `/etc/cni/net.d`). </del> <ins> Once the Kubelet is installed and communicating with the Kubernetes API Server, a CNI plugin can be installed using the instructions for that plugin. </ins> ## {{% heading "whatsnext" %}}