Kubernetes/Remove a service

All steps below assume you want to remove a service named service-foo from the clusters of the main (wikikube) group.
The ml-serve and dse-k8s groups are modelled closely on the main clusters, so many of the steps below are also applicable to these groups. Where there are specific differences they may be highlighted in the steps below.
Removal of Kubernetes Ingress or LVS is out of scope of this documentation for now.
Example of a removed service: T365742 
== 1. Destroy the service deployments ==
=== Staging ===
deploy1002:# cd /srv/deployment-charts/helmfile.d/services/service-foo
deploy1002:# helmfile -e staging -i destroyThe command above will show a diff of everything that will be removed, make sure that everything looks fine. Hit Yes to proceed.
=== Production ===
===== Note: Ensure you have silenced alarms for your service, and removed LVS if necessary =====
deploy1002:# cd /srv/deployment-charts/helmfile.d/services/service-foo
deploy1002:# helmfile -e codfw -i destroy
# All well?
deploy1002:# helmfile -e eqiad -i destroy
#
The service deployment is now destroyed.
== 2. Private data cleanup ==
=== Puppet: remove kubeconfig files ===
User accounts and tokens need to be removed from the deployment server 
* hieradata/common/profile/kubernetes/deployment_server.yaml 
** profile::kubernetes::deployment_server::services
profile::kubernetes::deployment_server::services:
 main:
 mathoid:
 usernames:
 - name: mathoid
 - name: mathoid-deploy
...
 service-foo:
+ ensure: absent
 usernames:
 - name: service-foo
 - name: service-foo-deploy
Example CR
* Once committed, merged and applied, this will remove the relevant directory and its files in the deployment server.
** Create a second CR that completely removes your service's stanza.
Example CR
=== Puppermaster secrets ===
Applications may have secrets (eg tokens, db user/pass etc), those are stored here
* hieradata/role/common/deployment_server/kubernetes.yaml
** profile::kubernetes::deployment_server_secrets::services
** Remove the relevant stanza
* same for labs/private repos
== 4. Remove the kubernetes service definitions ==
=== Preparing the commit ===
==== 1. Removing the namespace ====
Wikikube: 
* helmfile.d/admin_ng/values/main.yaml
** namespaces: 
** example
ml-serve, dse-k8s, and aux-k8s: 
* cluster groups are managed in their own files. 
==== 2. Remove your deployment ====
* git rm -rf helmfile.d/services/service-foo
* example
==== 3. Review your changes ====
Commit, and ask somebody from Service Ops to validate and merge. 
=== Deploy changes to helmfile.d/admin_ng ===
Someone from ServiceOps should be the one running the following: deploy1002:# sudo run-puppet-agent
deploy1002:# sudo -i 
deploy1002:# cd /srv/deployment-charts/helmfile.d/admin_ng/
deploy1002:# helmfile -e staging-codfw -i apply --context 5
# All well?
deploy1002:# helmfile -e staging-eqiad -i apply --context 5
deploy1002:# helmfile -e codfw -i apply --context 5
deploy1002:# helmfile -e eqiad -i apply --context 5Note: You should be seeing diffs here. If not, something is off! 
=== All Alright? ===
deploy1002:# sudo -i
deploy1002:# kube_env admin staging-codfw
deploy1002:# kubectl describe ns service-foo
Error from server (NotFound): namespaces "service-foo" not found
 
== 5. Remove the chart (if needed) ==
If your service was the last deployment of a chart, and it will not need to be reused, you can also remove the chart from the deployment-charts repo
Category:Kubernetes
Category:SRE Service Operations