Project: Litmus (Observability and Analysis / Chaos Engineering)
File: experiments.md

hide: toc <hr> The experiment takes a percentage of the total memory capacity of the Node. The helper pod runs on the target node to stress the resources of that node. So The experiment will not consume/hog the memory resources greater than the total memory available on Node. In other words there will always be an upper limit for the amount of memory to be consumed, which equal to the total available memory. Please refer to this for more details. Yes, ultimately these are different ways to simulate a degraded network. Both cases are expected to typically cause retransmissions (for tcp). The extent of degradation depends on the percentage of loss/corruption The experiment causes network degradation without the pod being marked unhealthy/unworthy of traffic by kube-proxy (unless you have a liveness probe of sorts that measures latency and restarts/crashes the container) The idea of this exp is to simulate issues within your pod-network OR microservice communication across services in different availability zones/regions etc.., Mitigation (in this case keep the timeout i.e., access latency low) could be via some middleware that can switch traffic based on some SLOs/perf parameters. If such an arrangement is not available - the next best thing would be to verify if such a degradation is highlighted via notification/alerts etc,. so the admin/SRE has the opportunity to investigate and fix things. Another utility of the test would be to see what the extent of impact caused to the end-user OR the last point in the app stack on account of degradation in access to a downstream/dependent microservice. Whether it is acceptable OR breaks the system to an unacceptable degree. The args passed to the tc netem command run against the target container changes depending on the type of n/w fault The pod cpu and memory chaos experiment till now (version 1.13.7) was using an exec mode of execution which means - we were execing inside the specified target container and launching process like `md5sum` and `dd` to consume the cpu and memory respectively. This is done by providing `CHAOSINJECTCOMMAND` and `CHAOS-KILL-COMMAND` in chaosengine CR. But we have some limitations of using this method. Those were: The chaos inject and kill command are highly dependent on the base image of the target container and may work for some and for others you may have to derive it manually and use it. For scratch images that don't expose shells we couldn't execute the chaos. To overcome this - The stress-chaos experiments (cpu, memory and io) are enhanced to use a non exec mode of chaos execution. It makes use of target container cgroup for the resource allocation and container pid namespace for showing the stress-ng process in target container. This `stress-ng` process will consume the resources on the target container without doing an exec. The new enhanced experiments are available from litmus 1.13.8 version. Precisely the role of the experiment. Cause n/w degradation w/o the pod being marked unhealthy/unworthy of traffic by kube-proxy (unless you have a liveness probe of sorts that measures latency and restarts/crashes the container) The idea of this exp is to simulate issues within your pod-network OR microservice communication across services in diff availability zones/regions etc.., Mitigation (in this case keep the timeout

access latency low) could be via some middleware that can switch traffic based on some SLOs/perf parameters. If such an arrangement is not available - the next best thing would be to verify if such a degradation is highlighted via notification/alerts etc,. so the admin/SRE has the opportunity to investigate and fix things. Another utility of the test would be to see what the extent of impact caused to the end-user OR the last point in the app stack on account of degradation in access to a downstream/dependent microservice. Whether it is acceptable OR breaks the system to an unacceptable degree The optional libs (like Pumba) is more of an illustration of how you can use 3rd party tools with litmus. Called the BYOC (Bring Your Own Chaos). The preferred LIB is `litmus`. To directly consume/manipulate the chaos resources (i.e., chaosexperiment, chaosengine or chaosresults) via API - you can directly use the kube API. The CRDs by default provide us with an API endpoint. You can use any generic client implementation (go/python are most used ones) to access them. In case you use go, there is a clientset available as well: Here are some simple CRUD ops against chaosresources you could construct with curl (I have used kubectl proxy, one could use an auth token instead)- just for illustration purposes. For example, assume this is the ```console curl -s http://localhost:8001/apis/litmuschaos.io/v1alpha1/namespaces/default/chaosengines -XPOST -H 'Content-Type: application/json' -d@pod-delete-chaosengine-trigger.json ``` ```console curl -s http://localhost:8001/apis/litmuschaos.io/v1alpha1/namespaces/default/chaosengines/nginx-chaos | jq '.status.engineStatus, .status.experiments[].verdict' ``` (say, this is the patch: https://gist.github.com/ksatchit/be54955a1f4231314797f25361ac488d) ```console curl --header "Content-Type: application/json-patch+json" --request PATCH --data '[{"op": "replace", "path": "/spec/engineState", "value": "stop"}]' http://localhost:8001/apis/litmuschaos.io/v1alpha1/namespaces/default/chaosengines/nginx-chaos ``` ```console curl -X DELETE localhost:8001/apis/litmuschaos.io/v1alpha1/namespaces/default/chaosengines/nginx-chaos \ -d '{"kind":"DeleteOptions","apiVersion":"v1","propagationPolicy":"Foreground"}' \ -H "Content-Type: application/json" ``` ```console curl -s http://localhost:8001/apis/litmuschaos.io/v1alpha1/namespaces/default/chaosresults/nginx-chaos-pod-delete | jq '.status.experimentStatus.verdict, .status.experimentStatus.probeSuccessPercentage' ``` There are some scenarios that can still occur despite whatever availability aids K8s provides. For example, take disk usage or CPU hogs -- problems you would generally refer to as "Noisy Neighbour" problems. Stressing the disk w/ continuous and heavy I/O for example can cause degradation in reads and writes performed by other microservices that use this shared disk - for example. (modern storage solutions for Kubernetes use the concept of storage pools out of which virtual volumes/devices are carved out). Another issue is the amount of scratch space eaten up on a node - leading to lack of space for newer containers to get scheduled (kubernetes too gives up by applying an "eviction" taint like "disk-pressure") and causes a wholesale movement of all pods to other nodes. Similarly w/ CPU chaos -- by injecting a rogue process into a target container, we starve the main microservice process (typically pid 1) of the resources allocated to it (where limits are defined) causing slowness in app traffic OR in other cases unrestrained use can cause node to exhaust resources leading to eviction of all pods. Yes. You can use either `the PODSAFFECTEDPERCENTAGE` or `TARGET_PODS` env to select multiple pods. Refer: . We can view the target pods from the experiment logs or inside chaos results. The `spec.definition.scope` & `.spec.definition.permissions` is mostly for indicative/illustration purposes (for external tools to identify and validate what are the permissions associated to run the exp). By itself, it doesn't influence how and where an exp can be

could remove these fields if needed (of course along w/ the crd validation) and store these manifests if desired. It will target the `[1:totalreplicas]`(based on PODSAFFECTEDPERC) numbers of random pods with matching labels(appinfo.applabel) and namespace(appinfo.appns). But if you want to target a specific pod then you can provide their names as a comma separated list inside `TARGETPODS`. Yes, you can provide service B details inside `DESTINATIONIPS` or `DESTINATIONHOSTS`. The `NETWORK_INTERFACE` should be `eth0`. The `NETWORK_INTERFACE` is the interface name inside the pod/container that needs to be targeted. You can find it by execing into the target pod and checking the available interfaces. You can try `ip link`, `iwconfig` , `ifconfig` depending on the tools installed in the pod either of those could work. The `SOCKET_PATH` by default takes the containerd socket path. If you are using something else like docker, crio or have a different socket path by any chance you can specify it. This is required to communicate with the container runtime of your cluster. In addition to this if container-runtime is different then provide the name of container runtime inside `CONTAINER_RUNTIME` ENV. It supports `docker`, `containerd`, and `crio` runtimes. The different ways are: Pod Chaos: `Appinfo`: Provide the target pod labels in the chaos engine appinfo section. `TARGET_PODS`: You can provide the target pod names as a Comma Separated Variable. Like pod1,pod2. Node Chaos: `TARGETNODE` or `TARGETNODES`: Provide the target node or nodes in these envs. `NODE_LABEL`: Provide the label of the target nodes. Yes, it selects the random pods based on the `PODAFFACTEDPERC` ENV. In pod-delete experiment it selects random pods for each iterations of chaos. But for rest of the experiments(if it supports iterations) then it will select random pods once and use the same set of pods for remaining iterations. We can use the Chaos exporter metrics for the same. One can also visualise these events along with time in chaos engine events. The MTTR can be validated by using statusCheck Timeout in the chaos engine. By default its value will be 180 seconds. We can also overwrite this using ChaosEngine. For more details refer The ramp time is the time duration to wait before and after injection of chaos in seconds. While the chaos interval is the time interval (in second) between successive chaos iterations. The appkind as pod is not supported explicitly. The supported appkind are `deployment`, `statefulset`, `replicaset`, `daemonset`, `rollout`, and `deploymentconfig`. But we can target the pods by following ways: provide labels and namespace at `spec.appinfo.applabel` and `spec.appinfo.appns` respectively and provide `spec.appinfo.appkind` as empty. provide pod names at `TARGET_PODS` ENV and provide `spec.appinfo` as nil `NOTE`: The `annotationCheck` should be provided as false Litmus broadly defines Kubernetes chaos experiments into two categories: application or pod-level chaos experiments and platform or infra-level chaos experiments. The former includes pod-delete, container-kill, pod-cpu-hog, pod-network-loss etc., while the latter includes node-drain, disk-loss, node-cpu-hog etc., The infra chaos experiments typically have a higher blast radius and impact more than one application deployed on the Kubernetes cluster. Litmus also categorizes experiments on the basis of the applications, with the experiments consisting of app-specific health checks. For a full list of supported chaos experiments, visit: https://hub.litmuschaos.io By default, the Litmus operator uses the litmus serviceaccount that is bound to a ClusterRole, in order to watch for the ChaosEngine resource across

However, the experiments themselves are associated with chaosServiceAccounts which are created by the developers with bare-minimum permissions necessary to execute the experiment in question. Visit the repo to view the experiment-specific rbac permissions. For example, here are the for container-kill chaos. The chaos CRs (chaosexperiment, chaosengine, chaosresults) themselves are namespace scoped and are installed in the same namespace as that of the target application. While most of the experiments can be executed with service accounts mapped to namespaced roles, some infra chaos experiments typically perform health checks of applications across namespaces & therefore need their serviceaccounts mapped to ClusterRoles. Litmus has a low entry barrier and is easy to install/use. Typically, it involves installing the chaos-operator, chaos experiment CRs from the , annotating an application for chaos and creating a chaosengine CR to map your application instance with a desired chaos experiment. Refer to the documentation to learn more on how to run a simple chaos experiment. The results of a chaos experiment can be obtained from the verdict property of the chaosresult custom resource. If the verdict is Pass, it means that the application under test is resilient to the chaos injected. Alternatively, Fail reflects that the application is not resilient enough to the injected chaos, and indicates the need for a relook into the deployment sanity or possible application bugs/issues. ``` kubectl describe chaosresult <chaosengine-name>-<chaos-experiment> -n <namespace> ``` The status of the experiment can also be gauged by the status property of the ChaosEngine. ``` Kubectl describe chaosengine <chaosengne-name> -n <namespace> ``` The chaos experiment (triggered after creation of the ChaosEngine resource) workflow consists of launching the chaos-runner pod, which is an umbrella executor of different chaos experiments listed in the engine. The chaos-runner creates one pod (job) per each experiment to run the actual experiment business logic, and also manages the lifecycle of these experiment pods (performs functions such as experiment dependencies validation, job cleanup, patching of status back into ChaosEngine etc.,). Optionally, a monitor pod is created to export the chaos metrics. Together, these 3 pods are a standard set created upon execution of the experiment. The experiment job, in turn may spawn dependent (helper) resources if necessary to run the experiments, but this depends on the experiment selected, chaos libraries chosen etc., Typically applications are expected to be annotated with litmuschaos.io/chaos="true" to lend themselves to chaos. This is in order to support selection of the right applications with similar labels in a namespaces, thereby isolating the application under test (AUT) & reduce the blast radius. It is also helpful for supporting automated execution (say, via cron) as a background service. However, in cases where the app deployment specifications are sacrosanct and not expected to be modified, or in cases where annotating a single application for chaos when the experiment itself is known to have a higher blast radius doesnt make sense (ex: infra chaos), the annotationCheck can be disabled via the ChaosEngine tunable annotationCheck (`.spec.annotationCheck: false`). Currently Litmus allows you to set your own/custom keys for Annotation filters, the value being true/false. To use your custom annotation, add this key under an ENV named as CUSTOM_ANNOTATION in ChaosOperator

A sample chaos-operator deployment spec is provided here for reference: ??? info "view the manifest" ``` apiVersion: apps/v1 kind: Deployment metadata: name: chaos-operator-ce namespace: litmus spec: replicas: 1 selector: matchLabels: name: chaos-operator template: metadata: labels: name: chaos-operator spec: serviceAccountName: litmus containers: name: chaos-operator image: litmuschaos/chaos-operator:latest command: chaos-operator imagePullPolicy: Always env: name: CUSTOM_ANNOTATION value: "mayadata.io/chaos" name: CHAOSRUNNERIMAGE value: "litmuschaos/chaos-runner:latest" name: WATCH_NAMESPACE value: name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name name: OPERATOR_NAME value: "chaos-operator" ``` Yes. As of today, the chaos resources are expected to co-exist in the same namespace, which typically is also the application's (AUT) namespace. The chaos logs can be viewed in the following manner. To view the successful launch/removal of chaos resources upon engine creation, for identification of application under test (AUT) etc., view the chaos-operator logs: ``` kubectl logs -f <chaos-operator-(hash)-(hash)> -n <chaos_namespace> ``` To view lifecycle management logs of a given (or set of) chaos experiments, view the chaos-runner logs: ``` kubectl logs -f <chaosenginename>-runner -n <chaosnamespace> ``` To view the chaos logs itself (details of experiment chaos injection, application health checks et al), view the experiment pod logs: ``` kubectl logs -f <experimentname(hash)(hash)> -n <chaosnamespace> ``` The chaos-operator generates Kubernetes events to signify the creation of removal of chaos resources over the course of a chaos experiment, which can be obtained by running the following command: ``` kubectl describe chaosengine <chaosengine-name> -n <namespace> ``` `Note:` Efforts are underway to add more events around chaos injection in subsequent releases. A chaos experiment can be stopped/aborted inflight by patching the .spec.engineState property of the chaosengine to stop . This will delete all the chaos resources associated with the engine/experiment at once. ``` kubectl patch chaosengine <chaosengine-name> -n <namespace> --type merge --patch '{"spec":{"engineState":"stop"}}' ``` The same effect will be caused by deleting the respective chaosengine resource. Once stopped/aborted, patching the chaosengine .spec.engineState with active causes the experiment to be re-executed. Another way is to re-apply the ChaosEngine YAML, this will delete all stale chaos resources, and restart ChaosEngine lifecycle. However, support is yet to be added for saving state and resuming an in-flight experiment (i.e., execute pending iterations etc.,) ``` kubectl patch chaosengine <chaosengine-name> -n <namespace> --type merge --patch '{"spec":{"engineState":"active"}}' ``` To restart chaosengine, check the .spec.engineState, which should be equal to stop, which means your chaosengine has gracefully completed, or forcefully aborted. In this case, restart is quite easy, as you can re-apply the chaosengine YAML to restart it. This will remove all stale chaos resources linked to this chaosengine, and restart its own lifecycle. Litmus provides a basic set of prometheus metrics indicating the total count of chaos experiments, passed/failed experiments and individual status of experiments specified in the ChaosEngine, which can be queried against the monitor pod. Work to enhance and improve this is underway. By default, the installation count of chaos-operator & run count of a given chaos experiment is collected as part of general analytics to gauge user adoption & chaos trends. However, if you wish to inhibit this, please use the following ENV setting on the chaos-operator deployment: ``` env: name: ANALYTICS value: 'FALSE' ``` Only one should be chosen ideally between minChaosInterval and instanceCount. However if both are specified minChaosInterval will be given priority. minChaosInterval specifies the minimum interval that should be present between the launch of 2 chaosengines and instanceCount specifies the exact number of chaosengines to be launched between the range (start and end time). SO we can choose depending on our requirements.