Portal:Toolforge/Admin/Kubernetes/Networking and ingress

This page describes the design, topology and setup of Toolforge's networking and ingress, specifically those bits related to webservices in the new kubernetes cluster.
= Network topology =
The following diagram is used to better describe the network topology and the different components involved.
When an user visits a webservice hosted in Toolforge, the following happens:
# nginx+lua: The first proxys is what we know as dynamicproxy. The nginx+lua setup knows how to send requests to the legacy webservices grid. There is a fall-through route for the k8s cluster. This proxy provides SSL termination for toolforge.org. There is a DNS A record pointing to the floating IP associated with the active proxy VM.
# haproxy: Then, haproxy knows which actual ingress worker nodes are alive in the k8s cluster. Proxy for both the k8s api-server on tcp/6443 and the web ingress on tcp/30000. There is a DNS A record with name k8s.tools.eqiad1.wikimedia.cloud pointing to the IP of this VM.
# nginx-ingress-svc: There is a nginx-ingress service of type NodePort, which means every ingress worker node listens on tcp/30000 and direct request to the nginx-ingress pod.
# nginx-ingress pod: The nginx-ingress pod will use ingress objects to direct the request to the appropriate service, but ingress objects need to exists beforehand.
# api-server: Ingress objects are created using the k8s API served by the api-server. They are automatically created using the webservice command, and the k8s API allows users to create and customize them too.
# ingress-admission-controller: There is a custom admission controller webhook that validates ingress config objects to enforce valid configurations.
# ingress object: After the webhook, the ingress objects are added to the cluster and the nginx-ingress pod can consume them.
# tool svc: The ingress object contained a mapping between URL/path and a service. The request is now in this tool specific service, which knows how to finally direct the query to the actual tool pod.
# tool pod: The request finally arrives at the actual tool pod.
= Components =
This section contains specific information about the different components involved in this setup.
There are mainly 2 different kinds of elements: those running outside kubernetes and those running inside.
== outside kubernetes ==
Information about components running outside kubernetes.
=== dynamicproxy ===
: See Toolforge dynamicproxy
This component is described in another wikitech page.
=== haproxy ===
This setup is fairly simple, deployed by puppet using the role role::wmcs::toolforge::k8s::haproxy.
We have a DNS name k8s.tools.eqiad1.wikimedia.cloud with an A record pointing to the active VM.
There should be a couple of VMs in a cold-standby setup (only one is actually working).
The haproxy configuration involves 2 ports, the "virtual servers":
* 6443/tcp for the main kubernetes api-server
* 30000/tcp for the ingress
Each "virtual server" has several backends:
* in the case of the api-server, backends are the controller nodes.
* in the case of the ingress, backends are ingress worker nodes.
== inside kubernetes ==
Explanation of the different components inside kubernetes.
=== calico ===
We use calico as the network overlay inside kubernetes. There is not a lot to say here, since we mostly use the default calico configuration. We only specify the CIDR of the pod network.
There is a single yaml file containing all the configuration, and deployed by puppet.
This file is modules/toolforge/templates/k8s/calico.yaml.erb in the puppet tree and /etc/kubernetes/calico.yaml in the final control nodes.
To load (or refresh) the configuration inside the cluster, use:
root@k8s-control-1:~# kubectl apply -f /etc/kubernetes/calico.yaml 
configmap/calico-config unchanged
customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org unchanged
customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org unchanged
customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org unchanged
customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org unchanged
customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org unchanged
customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org unchanged
customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org unchanged
customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org unchanged
customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org unchanged
customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org unchanged
customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org unchanged
customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org unchanged
customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org unchanged
customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org unchanged
clusterrole.rbac.authorization.k8s.io/calico-kube-controllers unchanged
clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers unchanged
clusterrole.rbac.authorization.k8s.io/calico-node unchanged
clusterrolebinding.rbac.authorization.k8s.io/calico-node unchanged
daemonset.apps/calico-node configured
serviceaccount/calico-node unchanged
deployment.apps/calico-kube-controllers unchanged
serviceaccount/calico-kube-controllers unchanged
Some things to take into account:
* Mind that all configuration is based on a specific version of calico. By the time of this writting, the version is v3.8.
=== nginx-ingress ===
We use nginx to process the ingress configuration in the k8s cluster.
Worth mentioning that we are using the comunity-based one (https://kubernetes.github.io/ingress-nginx/) rather than the NGINX Inc. one (https://github.com/nginxinc/kubernetes-ingress).
Starting from ingress-nginx 0.46.0 we deploy it using Helm. Helm is installed on the control plane nodes and uses the current users kubeconfig file to authenticate.
To deploy the component you can follow the standard toolforge procedure Portal:Toolforge/Admin/Kubernetes#Deploy_new_version.
==== error handling ====
We have a tool called fourohfour which is set as the default backend for nginx-ingress. This tool presents an user friendly 404 page.
=== ingress objects ===
Ingress objects can be created in 2 ways:
* directly using the kubernetes API
* by using the webservice command.
Objects have this layout. Toolforge tool name fourohfour is used as example:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
 annotations:
 nginx.ingress.kubernetes.io/configuration-snippet: |
 rewrite ^(/fourohfour)$ $1/ redirect;
 nginx.ingress.kubernetes.io/rewrite-target: /fourohfour/$2
 labels:
 name: fourohfour
 toolforge: tool
 tools.wmflabs.org/webservice: "true"
 tools.wmflabs.org/webservice-version: "1"
 name: fourohfour
 namespace: tool-fourohfour
spec:
 rules:
 - host: toolsbeta.wmflabs.org
 http:
 paths:
 - backend:
 serviceName: fourohfour
 servicePort: 8000
 path: /fourohfour(/|$)(.*)
status:
 loadBalancer: {}
NOTE: the rewrite/redirect configuration is really important and is part of the behaviour users expect. See phabricator ticket T242719 for example.
This ingress object is pointing to a service which should have this layout:
apiVersion: v1
kind: Service
metadata:
 labels:
 name: fourohfour
 toolforge: tool
 tools.wmflabs.org/webservice: "true"
 tools.wmflabs.org/webservice-version: "1"
 name: fourohfour
 namespace: tool-fourohfour
spec:
 clusterIP: x.x.x.x
 ports:
 - name: http
 port: 8000
 protocol: TCP
 targetPort: 8000
 selector:
 name: fourohfour
 sessionAffinity: None
 type: ClusterIP
status:
 loadBalancer: {}
Note that both objects are namespaced to the concrete tool namespace.
=== ingress admission controller ===
This k8s API webhook checks ingress objects before they are accepted by the API itself.
It enforces (or prevents) ingress configurations that may produce malfunctioning in the webservices running in kubernetes, like pointing URLs/path to tools that are not ready to handle them.
The code is written in golang, and can be found here:
* https://gerrit.wikimedia.org/r/admin/projects/cloud/toolforge/ingress-admission-controller
* https://github.com/wikimedia/cloud-toolforge-ingress-admission-controller
See also Portal:Toolforge/Admin/Kubernetes/Custom_components.
= How to test the setup =
See logs for nginx-ingress:
root@tools-k8s-control-3:~# kubectl logs -lapp.kubernetes.io/name=ingress-nginx -n ingress-nginx
192.168.50.0 - [192.168.50.0] - - [18/Dec/2019:12:02:21 +0000] "GET /potd-feed/potd.php/commons/potd-400x300.rss HTTP/1.1" 503 2261 "-" "FeedFetcher-Google; (+http://www.google.com/feedfetcher.html)" 402 0.003 [upstream-default-backend] [] 192.168.15.133:8000 2261 0.000 503 2d08390256b1629bc60552710e4b47e1
192.168.34.128 - [192.168.34.128] - - [18/Dec/2019:12:02:22 +0000] "GET /favicon.ico HTTP/1.1" 404 1093 "https://tools.wmflabs.org/wikisense/Gallery.php?&wikifam=commons.wikimedia.org&img_user_text=Hoanglong2807" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) coc_coc_browser/83.0.144 Chrome/77.0.3865.144 Safari/537.36" 576 0.529 [upstream-default-backend] [] 192.168.15.133:8000 2116 0.528 404 5fc83ada234e3a5a5da5e42a6583f992
192.168.15.128 - [192.168.15.128] - - [18/Dec/2019:12:02:23 +0000] "GET /mediawiki-feeds/ HTTP/1.1" 503 2236 "https://tools.wmflabs.org/mediawiki-feeds/" "Mozilla/5.0+(compatible; UptimeRobot/2.0; http://www.uptimerobot.com/)" 487 0.003 [upstream-default-backend] [] 192.168.15.133:8000 2236 0.004 503 e0341d943aba393e30fc692151da0790
[..]
TODO: extend this. Perhaps refactor into its own wiki page?
= See also =
* Toolforge k8s RBAC and PodSecurityPolicy -- documentation page
* phabricator T228500 - Toolforge: evaluate ingress mechanism -- original ticket to design the ingress (epic)
* phabricator T234037 - Toolforge ingress: decide on final layout of north-south proxy setup -- original ticket to design the outer proxy layout
* phabricator T234231 - Toolforge ingress: decide on how ingress configuration objects will be managed -- original ticket to design ingress object management
* phabricator T235252 - Toolforge: SSL support for new domain toolforge.org -- original ticket to handle SSL support for the domain toolforge.org and the new k8s cluster in general
* phabricator T234617 - Toolforge. introduce new domain toolforge.org -- original ticket to introduce the domain toolforge.org
* phabricator T234032 - Toolforge ingress: create a default landing page for unknown/default URLs -- about the fourohfour tool
* Resolution on new domains usage for WMCS
* Deploying Toolforge kubernetes
* Legacy redirector for webservices