Kubernetes/Deployment Charts

= Deployment Charts =
All of our existing Kubernetes clusters make use of a single git repository located at operations/deployment-charts:
This repository is organized as follows:
* Helm charts in /charts
* Helmfile deployments in /helmfile.d
* Common templates in /_scaffold
* Template modules in /modules
* JSON schemas of all CustomResourceDefinition objects in /jsonschema
* Any custom deployments that cannot use helm for any reason, in /custom_deploy.d
* Supporting test files and scripts
See the README file in the repository for more detailed information.
== Helm Charts ==
A Helm chart is a set of files that describes a packaging together of Kubernetes resources so that they can be easily deployed as a unit and updated programmatically.
Once committed to the master branch, our helm charts are automatically published to ChartMuseum.
== Common Templates ==
We have developed a set of common templates for our helm charts.
== Enabling egress to services external to Kubernetes ==
=== Install the external-services helm template ===
==== In a new chart ====
The scaffolding should inject the necessary templates automatically. No need to do anything. 
==== In an existing chart ====
Note: As a prerequisite, you will need to have sextant installed.
1. In order to enable egress to services living outside of Kubernetes (such as Kafka, Presto, Kerberos, etc), add "base.external-services-networkpolicy:1.0", to your chart's package.json file. 
Run the following command to update the chart dependency lockfile, as well as install the base/external-services-networkpolicy template in your chart templates/vendor directory:
# Assuming you're working on charts/superset
$ cd path/to/deployment-charts
$ sextant --modulepath modules vendor charts/superset
INFO:sextant:Copied modules/base/external-services-networkpolicy_1.0.1.tpl => charts/superset/templates/vendor/base/external-services-networkpolicy_1.0.1.tpl
2. Add the following line at the end of your chart's templates/networkpolicy.yaml file
{{ include "base.networkpolicy.egress.external-services" . }}
Note: make sure this include isn't specified in a range loop, as it will lead to duplicate NetworkPolicy resources, which will cause your deployment to fail in cryptic ways.
3. Add the following lines in your chart's values.yaml file
# The set of external services to allow egress to
# Example:
# kafka:
# - main-codfw
# - main-eqiad
# presto:
# - analytics
#
# See https://wikitech.wikimedia.org/wiki/Kubernetes/Deployment_Charts#Enabling_egress_to_services_external_to_Kubernetes
# for the list of supported services
external_services: {}
4. Finally, bump your chart version in your chart's Chart.yaml file.
=== Select what services you want to enable egress to ===
Add the following structure to your chart or release values, by adapting the values associated with each service type:
# Note: only include the services you actually need, and not *all* of them.
# We mainly include them here to help you figure out what's available.
external_services:
 presto: [analytics]
 kerberos: [kdc]
 cas: [idp]
 druid: [analytics, public]
 redis-6379: [misc]
 kafka: [jumbo-eqiad]
 zookeeper: [flink-eqiad]
 cassandra: [ml-cache-a-eqiad]
 hadoop-worker: [analytics]
 hadoop-master: [analytics]
 mariadb: [analytics-meta]
 postgresql: [analytics]
The NetworkPolicy resource(s) allowing egress traffic to reach the specified services will be created next time you redeploy your application.
Note: all exposed service types and associated instances are listed in Kubernetes. Connect to the deployment server, and use kubectl to list all services in the external-services namespace.
brouberol@deploy1002:~$ kubectl get services -n external-services -o jsonpath='{.items[*].metadata.name}' | tr ' ' '\n'
cas-idp
cas-idp-test
cassandra-analytics-query-service-storage-a-codfw
cassandra-analytics-query-service-storage-a-eqiad
cassandra-analytics-query-service-storage-b-codfw
cassandra-analytics-query-service-storage-b-eqiad
cassandra-cassandra-dev-a-codfw
cassandra-cassandra-dev-b-codfw
cassandra-ml-cache-a-codfw
cassandra-ml-cache-a-eqiad
cassandra-restbase-a-codfw
cassandra-restbase-a-eqiad
cassandra-restbase-b-codfw
cassandra-restbase-b-eqiad
cassandra-restbase-c-codfw
cassandra-restbase-c-eqiad
cassandra-sessions-a-codfw
cassandra-sessions-a-eqiad
druid-analytics
druid-analytics-test
druid-public
elasticsearch-cloudelastic-chi-eqiad-cloudelastic
elasticsearch-cloudelastic-omega-eqiad-cloudelastic
elasticsearch-cloudelastic-psi-eqiad-cloudelastic
elasticsearch-production-search-codfw-cirrus
elasticsearch-production-search-eqiad-cirrus
elasticsearch-production-search-omega-codfw-cirrus
elasticsearch-production-search-omega-eqiad-cirrus
elasticsearch-production-search-psi-codfw-cirrus
elasticsearch-production-search-psi-eqiad-cirrus
elasticsearch-relforge-eqiad-relforge
elasticsearch-relforge-eqiad-small-alpha-relforge
hadoop-master-analytics
hadoop-master-analytics-test
hadoop-worker-analytics
hadoop-worker-analytics-test
kafka-jumbo-eqiad
kafka-logging-codfw
kafka-logging-eqiad
kafka-main-codfw
kafka-main-eqiad
kafka-test-eqiad
kerberos-kdc
mariadb-analytics-meta
opensearch-datahubsearch
postgresql-analytics
presto-analytics
presto-analytics-test
redis-6378-misc
redis-6379-misc
redis-6380-misc
redis-6381-misc
redis-6382-misc
zookeeper-analytics-eqiad
zookeeper-druid-analytics-eqiad
zookeeper-druid-analytics-test-eqiad
zookeeper-druid-public-eqiad
zookeeper-flink-codfw
zookeeper-flink-eqiad
zookeeper-main-codfw
zookeeper-main-eqiad
== Testing a Chart ==
Our CI will run a set of tests to validate a chart. In order to do that locally you will need:
* Python 3.9 or higher
* python3-yaml linux package or PyYAML (via pip)
* Ruby 3.0 or higher
* rake, a ruby make-like utility
* helm3
* docker or another container engine 
To ensure the validity of our templates, we use two sets of fixtures. 
* fixtures under charts/mychart/.fixtures for various test cases
* fixtures under .fixtures providing general data (eg mariadb sections, kafka_brokers etc) for misc enviroments (staging, eqiad, etc)
** run rake refresh_fixtures to get the latest ones from our puppet repo.
==== Quick tests ====
To quickly test against a specific fixture:
 helm template \ 
 --values .fixtures/general-eqiad.yaml \
 -f charts/mychart/.fixtures/tls_enabled.yaml \
 charts/mychart \
 --debug
==== CI tests ====
To Validate against our CI:
 rake run_locally["check_charts[lint/validate\,mychart]"]
Category:Kubernetes
Category:SRE Service Operations