Project: Flannel (Runtime / Cloud Native Network)
File: running.md

Once you have pushed configuration JSON to `etcd`, you can start `flanneld`. If you published your config at the default location, you can start `flanneld` with no arguments. Flannel will acquire a subnet lease, configure its routes based on other leases in the overlay network and start routing packets. It will also monitor `etcd` for new members of the network and adjust the routes accordingly. After flannel has acquired the subnet and configured backend, it will write out an environment variable file (`/run/flannel/subnet.env` by default) with subnet address and MTU that it supports. For more information on checking the IP range for a specific host, see . Flanneld does not support running multiple networks from a single daemon (it did previously as an experimental feature). However, it does support running multiple daemons on the same host with different configurations. The `-subnet-file` and `-etcd-prefix` options should be used to "namespace" the different daemons. For example ``` flanneld -subnet-file /vxlan.env -etcd-prefix=/vxlan/network ``` Download a `flannel` binary. ```bash wget https://github.com/flannel-io/flannel/releases/latest/download/flanneld-amd64 && chmod +x flanneld-amd64 ``` Run the binary. ```bash sudo ./flanneld-amd64 # it will hang waiting to talk to etcd ``` Run `etcd`. Follow the instructions on the , or, if you have docker just do ```bash docker run --rm --net=host quay.io/coreos/etcd ``` Observe that `flannel` can now talk to `etcd`, but can't find any config. So write some config. Either get `etcdctl` from the , or use `docker` again. ```bash docker run --rm -e ETCDCTL_API=3 --net=host quay.io/coreos/etcd etcdctl put /coreos.com/network/config '{ "Network": "10.5.0.0/16", "Backend": {"Type": "vxlan"}}' ``` Now `flannel` is running, it has created a VXLAN tunnel device on the host and written a subnet config file ```bash cat /run/flannel/subnet.env FLANNEL_NETWORK=10.5.0.0/16 FLANNEL_SUBNET=10.5.72.1/24 FLANNEL_MTU=1450 FLANNEL_IPMASQ=false ``` Each time flannel is restarted, it will attempt to access the `FLANNEL_SUBNET` value written in this subnet config file. This prevents each host from needing to update its network information in case a host is unable to renew its lease before it expires (e.g. a host was restarting during the time flannel would normally renew its lease). The `FLANNELSUBNET` value is also only used if it is valid for the etcd network config. For instance, a `FLANNELSUBNET` value of `10.5.72.1/24` will not be used if the etcd network value is set to `10.6.0.0/16` since it is not within that network range. Subnet config value is `10.5.72.1/24` ```bash cat /run/flannel/subnet.env FLANNEL_NETWORK=10.5.0.0/16 FLANNEL_SUBNET=10.5.72.1/24 FLANNEL_MTU=1450 FLANNEL_IPMASQ=false ``` etcd network value is `10.6.0.0/16`. Since `10.5.72.1/24` is outside of this network, a new lease will be allocated. ```bash export ETCDCTL_API=3 etcdctl get

{ "Network": "10.6.0.0/16", "Backend": {"Type": "vxlan"}} ``` Flannel uses the interface selected to register itself in the datastore. The important options are: `-iface string`: Interface to use (IP or name) for inter-host communication. `-public-ip string`: IP accessible by other nodes for inter-host communication. The combination of the defaults, the autodetection and these two flags ultimately result in the following being determined: An interface (used for MTU detection and selecting the VTEP MAC in VXLAN). An IP address for that interface. A public IP that can be used for reaching this node. In `host-gw` it should match the interface address. Please be aware of the following flannel runtime limitations. The datastore type cannot be changed. The backend type cannot be changed. (It can be changed if you stop all workloads and restart all flannel daemons.) You can change the subnetlen/subnetmin/subnetmax with a daemon restart. (Subnets can be changed with caution. If pods are already using IP addresses outside the new range they will stop working.) The clusterwide network range cannot be changed (without downtime). Docker daemon accepts `--bip` argument to configure the subnet of the docker0 bridge. It also accepts `--mtu` to set the MTU for docker0 and veth devices that it will be creating. Because flannel writes out the acquired subnet and MTU values into a file, the script starting Docker can source in the values and pass them to Docker daemon: ```bash source /run/flannel/subnet.env docker daemon --bip=${FLANNELSUBNET} --mtu=${FLANNELMTU} & ``` Systemd users can use `EnvironmentFile` directive in the `.service` file to pull in `/run/flannel/subnet.env` If you want to leave default docker0 network as it is and instead create a new network that will be using flannel you do so like this: ```bash source /run/flannel/subnet.env docker network create --attachable=true --subnet=${FLANNELSUBNET} -o "com.docker.network.driver.mtu"=${FLANNELMTU} flannel ``` Vagrant has a tendency to give the default interface (one with the default route) a non-unique IP (often 10.0.2.15). This causes flannel to register multiple nodes with the same IP. To work around this issue, use `--iface` option to specify the interface that has a unique IP. When running with a backend other than `udp`, the kernel is providing the data path with `flanneld` acting as the control plane. As such, `flanneld` can be restarted (even to do an upgrade) without disturbing existing flows. However in the case of `vxlan` backend, this needs to be done within a few seconds as ARP entries can start to timeout requiring the flannel daemon to refresh them. Also, to avoid interruptions during restart, the configuration must not be changed (e.g. VNI, --iface values).