PAWS/Tools/Admin/Chico's notes

== Notes on setting up a PAWS staging env in toolsbeta VPS project (T188428)==
Using https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/ as docs
* Doesn't seem like PAWS is properly puppetized
** I see the apt-pinning defnitions, but not where tools-paws-master-01 installs needed packages (k8s, docker, etc)
* Docker and k8s repos are defined for Xenial, though tools-paws-master-01 is stretch
** Docker does have current stretch versions, k8s does not (left docker as strech, k8s as xenial)
* Unsure about cgroup driver used by docker. Official docs says to place { "exec-opts": ["native.cgroupdriver=systemd"] } in /etc/docker/daemon.json since the prod version does not have that I'm ommiting it for now
* swap needs to be turned off for docker, done manually
* started the k8s cluster with flannel
** kubeadm init --pod-network-cidr=10.244.0.0/16
* Allow user chicocvenancio to use k8s
** chicocvenancio@toolsbeta-paws-master-01:~$ mkdir -p $HOME/.kube
** chicocvenancio@toolsbeta-paws-master-01:~$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
** chicocvenancio@toolsbeta-paws-master-01:~$ sudo chown $(id -u):$(id -g) $HOME/.kube/config
* Docs say to set /proc/sys/net/bridge/bridge-nf-call-iptables to 1, it already was 1
* get flannel pods (docs mentions v0.9.1 I used latest version)
** kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/v0.10.0/Documentation/kube-flannel.yml
* exported Yuvi's git-crypt key
** yuvipanda@tools-paws-master-01:~/paws$ sudo -E git-crypt export-key /tmp/paws-key
** chicocvenancio@tools-paws-master-01:~$ sudo chown chicocvenancio /tmp/paws-key
** chicocvenancio@tools-paws-master-01:~$ scp /tmp/paws-key toolsbeta-paws-master-01.toolsbeta:~/paws-key
* Intalled git-crypt
** chicocvenancio@toolsbeta-paws-master-01:~$ sudo apt-get install git-crypt
* ran into Error: could not find tiller
**fixed with `helm init`
* Tiller won't start due to a lack of nodes
* Create new node
** Is there a way to adhere to naming convention when using instance count in horizon?
** toolsbeta-paws-worker-1001
*** Since its not puppetized, going for manual again
**Node joining brings up tiller
* Once tiller is up we can run "sudo ./build.py deploy prod --install" to install PAWS
**We need to fix two things Yuvi did CLI and did not push to repo (yuvi's .bash_history invaluable to get these right)
*** Tiller RBAC
**** Done now by setting a non ideal permissive clusterrolebinding
*****chicocvenancio@toolsbeta-paws-master-01:~$ kubectl create clusterrolebinding permissive-binding --clusterrole=cluster-admin --user=admin --user=kubelet --group=system:serviceaccounts
*** the hub_db pvc is not defined at all in the repo, found the definition in yuvi's .bash_history
**** chicocvenancio@toolsbeta-paws-master-01:~$ kubectl -n prod apply -f /mnt/nfs/labstore-secondary-home/yuvipanda/paw-c/hub-pv.yaml
* TODO: setup a new OAuth consumer for PAWS-beta
** Waiting aproval https://meta.wikimedia.org/w/index.php?title=Special:OAuthListConsumers/view/bb705f2027fbdb8fc434d4fdf9a7c482&name=PAWS-beta&publisher=&stage=0
* TODO: stop these annoying pre-puller deamonsets
** This is actually not annoying and good once I pointed them to working docker repositories
* Right now paws-beta uses a completely different way (WMCS-wise) for traffic ingress, I thought this simpler than copying the paws-proxy instance, in fact we can probably drop those instances (VPS cloud project) and improve production after some testing
** I did not get the ideal k8s LoadBalancer service to work with external IPs, instead I used a NodePort service and pointed a webproxy to one of the nodes (any will do)
*** This does mean that if that node fails the site will proxy will fail, which is NOT ok for production
* Differences between PAWS-beta and prod:
** Already without the query-killer image
** Deploy-hook image uses artful and not zesty
** Per above, traffic ingress is different
== Notes on K8S upgrade in PAWS-beta ==
* Using docs https://kubernetes.io/docs/tasks/administer-cluster/kubeadm-upgrade-1-9/
=== cluster control plane ===
*Upgrade kubeadm
** curl -sSL https://dl.k8s.io/release/v1.9.4/bin/linux/amd64/kubeadm > /usr/bin/kubeadm
** chmod a+rx /usr/bin/kubeadm
*Verify kubeadm version
** chicocvenancio@toolsbeta-paws-master-01:~$ kubeadm version
** kubeadm version: &version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.4", GitCommit:"bee2d1505c4fe820744d26d41ecd3fdd4a3d6546", GitTreeState:"clean", BuildDate:"2018-03-12T16:21:35Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
*root@toolsbeta-paws-master-01:~# kubeadm upgrade plan
*kubeadm upgrade apply v1.9.4
=== Upgrade nodes ===
*drain each node
**chicocvenancio@toolsbeta-paws-master-01:~$ kubectl get nodes
** NAME STATUS ROLES AGE VERSION
** toolsbeta-paws-master-01 Ready master 13d v1.9.3
** toolsbeta-paws-worker-1001 Ready <none> 13d v1.9.3
** toolsbeta-paws-worker-1002 Ready <none> 13d v1.9.3
** toolsbeta-paws-worker-1003 Ready <none> 13d v1.9.3
** chicocvenancio@toolsbeta-paws-master-01:~$ kubectl drain toolsbeta-paws-master-01 --ignore-daemonsets
*upgrade k8s:
**sudo apt-get update
**sudo apt-get install kubeadm=1.9.4-00 kubectl=1.9.4-00 kubelet=1.9.4-00
*verify kubelet is running
**systemctl status kubelet
*Uncordon node and verify it is ready
**kubectl uncordon toolsbeta-paws-master-01
**kubectl get nodes
* Rinse, repeat for each node until all nodes in version v1.9.4:
**chicocvenancio@toolsbeta-paws-master-01:~$ kubectl get nodes
** NAME STATUS ROLES AGE VERSION
** toolsbeta-paws-master-01 Ready master 13d v1.9.4
** toolsbeta-paws-worker-1001 Ready <none> 13d v1.9.4
** toolsbeta-paws-worker-1002 Ready <none> 13d v1.9.4
** toolsbeta-paws-worker-1003 Ready <none> 13d v1.9.4
== k8s upgrade in PAWS ==
* Went through the same steps as beta, but there are more nodes, and ran into a few new issues
* apt-pinning by puppet
** This is done by raising a version's priority, to keep the workflow and guarantee the "drain => upgrade => uncordon => next node" order used "=1.9.4-00" to request specific version to apt-get install
*nodes taking a very long time to drain
** Google led me to https://medium.com/@felipedutratine/when-you-try-to-drain-a-kubernetes-node-but-it-blocks-5aba9592d7c9
*** get the pods on the node and delete the ones still there
**** kubectl get pods -o wide --all-namespaces|grep worker-1001
*****kube-system kube-flannel-ds-stbq6 1/1 Running 1 63d 10.68.23.135 tools-paws-worker-1001
*****kube-system kube-proxy-zzrj2 1/1 Running 0 3h 10.68.23.135 tools-paws-worker-1001
*****prod proxy-5cd7d56555-tm4p6 2/2 Running 0 20d 10.244.7.45 tools-paws-worker-1001
*****support support-nginx-ingress-controller-sbl64 1/1 Running 4 63d 10.68.23.135 tools-paws-worker-1001
**** Mind kube-flannel, support-nginx-ingress-controller, and kube-proxy are DeamonSet controlled, so they wouldn't interfere
***** chicocvenancio@tools-paws-master-01:~$ kubectl delete pod proxy-5cd7d56555-tm4p6 -n prod
* Created bash script to go through each one (/home/chicocvenancio/update_nodes.sh)
* Sent Change 419599 to pin k8s to version 1.9.4 in PAWS.