❝​ ​Before everything else, getting ready is the secret of success​ ​❞Henry Ford

Running Containers andKubernetes in Production
Checklist

Provided bywww.cloudroads.com

Congratulations on leaping onto the world of cloud-native and deciding to run containers
and Kubernetes in production. You have taken a strategic move that’s going to pay you
dividends down the road. Now might be a good time to plan your next vacation!
Cloud-native technologies are disruptive as they enable organizations to cut time to market,
lower IT costs, innovate rapidly, and run infrastructure at scale. However, cloud-nativetechnologies or any technology for that matter shouldn’t disrupt your business operations
in a negative way.
We put together checklists compiled from numerous production environments to make iteasier for you and your team to reduce the potential issues of running Container andKubernetes in production efficiently and securely.

Why Checklists?
There are many aspects of being successful adopting any new technology. For example, youhave to understand the technology itself, train your team, and develop the processes toenable a successful operation of the technology in your organization.
We couldn’t find a better tool than checklists to do just that . Checklists have been used for
decades in the aviation and healthcare industries to dramatically improve the odds ofsuccess and deliver consistent results especially under difficult circumstances.
To help you prepare and understand this challenge, CloudRoads compiled this helpful
multi-part checklist. Note that the following items represent a sample of 100 plus discrete
steps that CloudRoads routinely undertaken as part of an actual Kubenetes production
readiness exercise. To learn more, please contact us.
Checklist items include:​Platform
Build for resiliency and failure from the outset​ esource Management
RAvoid that noisy neighbor and grow your cluster​ perationsO
Aim for operational excellence from day one​ ecurity
SReduce the attack surface and be a minimalist​ overnanceG
Control your cluster before someone else does-1-Copyright © 2020 CloudRoads LLC. All Rights Reserved

Platform❏ Consider the different options to size the cluster appropriately (i.e: small, medium,large, and very large) and build a platform for the next 12-18 months. It’s always
better to start with many small workers nodes and scale later on as your workloads
start to grow 1​❏ Formulate your own Disaster Recovery Plan (DRP). Keep it simple and make sure it’sreviewed regularly❏ Document and test your DRP using drills accounting for a diverse set of scenarios.
❏ Schedule backups for the key components of the platform (i.e: etcd, kubernetes
objects,..etc)❏ Run End-to-End (E2E) conformance tests and carefully review the results. This will
make certain you’re running a certified distribution of Kubernetes regardless of thevendor or cloud provider 2​❏ Provide a highly-available routing point for incoming traffic (aka ingress).
❏ Determine and document the maintenance cycle for the platform (i.e: upgrades) and
its underlying infrastructure (i.e: OS patching)❏ Fully document the architecture of the environment including: hardware, software,and 3​rd​ party integrations❏ Compile, document, and address specific issues encountered during pre-production
rollout to make sure they won’t resurface in production-2-Copyright © 2020 CloudRoads LLC. All Rights Reserved

Resource Management❏ Separate workloads using namespaces and limit the compute, storage, and numberof objects for each namespace. This will prevent containers from running withunbounded resources and eventually consuming all the resources on a particularnode❏ Identify and document the parameters to scale the cluster as you onboard more
applications. At some point, you will have to decide whether to enlarge the existingcluster to accommodate for more workloads or build a new one❏ Monitor and measure the performance of the underlying infrastructure from dayone. This serves two purposes: To better understand your usage patterns over time ,and feed data into the capacity planning exercise in the future when you want todecide to enlarge the cluster or build a new one❏ Rotate the logs for the node, containerized applications, and container runtime.
Failing to do so might get you in a situation in which logs consume all the storage ona particular node
❏ Ensure unused images and killed containers do not take up unnecessary disk space.
This can be done by configuring out of resource handling in Kubernetes 3​❏ Limit the number of pods that can run on your worker nodes. This will make sure
you never overutilize the computational power in your cluster. Also, you can use thisas a metric to scale up or down worker nodes❏ Unless you’re working with mission-critical applications or a very large volume of
deployed applications, limit the number of requests handled by the Kube-API server.
This to ensure the stability of the manager nodes in terms of CPU and memory❏ Provide a range of detachable storage options (i.e: SSD, HHD,...etc) to be used by thefirst wave of applications on the platform. Review as you onboard more applications
to provide more options to the different types of workloads (stateless, stateful, ..etc)-3-Copyright © 2020 CloudRoads LLC. All Rights Reserved

Operations❏ Set up health checks (readiness and live probes) in your applications deployment
manifests. This will increase the overall reliability and availability of yourapplications❏ Use Pod Affinity and Anti-Affinity rules to co-locate or spread pods across data
centers or availability zones to increase the overall availability of your applications
and balance resource utilization❏ Use ​PodDistruptionBudgets​ to ensure the availability of the most criticalapplications on the platform. This will make sure that such applications can survive
voluntary eviction by the scheduler❏ Develop an operations playbook . This will define the roles and responsibilities ofeach one in your team to be able to respond swiftly using predetermined steps toresolve issues❏ Utilize ​PreStop​ hooks to gracefully shutdown pods. This will make sure yourapplications can properly release resources and connections when forced toterminate❏ Always use ​Deployment ​manifests to define application pods even if you’rerunning only one replica. This will make sure your applications are fault-tolerant.
❏ Automate the process of provisioning the underlying hosts. Failure is almost
inevitable in distributed systems and you have to be prepared to add, remove, or
replace nodes in the cluster quickly❏ Build Monitoring and Logging stacks that ensure both the health and functionalityof the platform and applications are monitored and measured. This can be done witha blend of whitebox & blackbox monitoring , the four golden signals, application
performance monitoring (APM), and more-4-Copyright © 2020 CloudRoads LLC. All Rights Reserved​Security❏ Reduce the attack surface by being a minimalist when it comes to the container
image file. Start with the smallest image possible and only add the libraries required
for the application to run❏ Run rootless containers. This can enforced at the pod level using the ​runAsUser
securityContext​ , or at the cluster level using a ​PodSecurityPolicy❏ Update or Patch the underlying host or virtual machine regularly. A container is aprocess and it’s as secure as its underlying host❏ Enable RBAC making sure it’s in the authorization mode ​--authorization-mode❏ Limit East-West (Intra-cluster) Traffic. By default, pods can communicate with eachother. You can implement a network policy using your CNI plugin to deny accessbetween namespaces by default❏ Use the least privileges principle across the board❏ Rotate encryption keys regularly❏ Limit access to the underlying infrastructure of the production platform to theadministration team.
❏ Identify , document, and fix Kubernetes misconfigurations using tools such asPolaris​ or ​Starboard❏ Regularly look for vulnerabilities by following the ​CIS guidelines​ using tools such as
kube-bench-5-Copyright © 2020 CloudRoads LLC. All Rights Reserved

Governance❏ Follow the CIS gudielines for hardening the underlying hosts, container runtime,and Kubernetes❏ Integrate into your CI/CD pipeline guardrails to identify possible vulnerabilities.
This includes container image scanning, static code analysis, using a linter, and CIS
benchmarks (link).
❏ Develop golden images (minimal, scanned, and ready-to-go images) for all yourdevelopment teams to limit the exposure to vulnerabilities from untrusted images.

❏ Ensure all data and secrets are encrypted at rest and in flight-6-Copyright © 2020 CloudRoads LLC. All Rights Reserved