Project: Litmus (Observability and Analysis / Chaos Engineering)
File: pod-delete.md

It Causes (forced/graceful) pod failure of specific/random replicas of an application resources. It tests deployment sanity (replica availability & uninterrupted service) and recovery workflow of the application !!! tip "Scenario: Deletes kubernetes pod" ??? info "View the uses of the experiment" In the distributed system like kubernetes it is very likely that your application replicas may not be sufficient to manage the traffic (indicated by SLIs) when some of the replicas are unavailable due to any failure (can be system or application) the application needs to meet the SLO(service level objectives) for this, we need to make sure that the applications have minimum number of available replicas. One of the common application failures is when the pressure on other replicas increases then to how the horizontal pod autoscaler scales based on observed resource utilization and also how much PV mount takes time upon rescheduling. The other important aspects to test are the MTTR for the application replica, re-elections of leader or follower like in kafka application the selection of broker leader, validating minimum quorum to run the application for example in applications like percona, resync/redistribution of data. This experiment helps to reproduce such a scenario with forced/graceful pod failure on specific or random replicas of an application resource and checks the deployment sanity (replica availability & uninterrupted service) and recovery workflow of the application. ??? info "Verify the prerequisites" Ensure that Kubernetes Version > 1.16 Ensure that the Litmus Chaos Operator is running by executing <code>kubectl get pods</code> in operator namespace (typically, <code>litmus</code>).If not, install from <a href="https://v1-docs.litmuschaos.io/docs/getstarted/#install-litmus">here</a> Ensure that the <code>pod-delete</code> experiment resource is available in the cluster by executing <code>kubectl get chaosexperiments</code> in the desired namespace. If not, install from <a href="https://hub.litmuschaos.io/api/chaos/master?file=faults/kubernetes/pod-delete/fault.yaml">here</a> ??? info "View the default validations" The application pods should be in running state before and after chaos injection. !!! tip "NOTE" If you are using this experiment as part of a litmus workflow scheduled constructed & executed from chaos-center, then you may be making use of the RBAC, which is pre installed in the cluster as part of the agent setup. ??? note "View the Minimal RBAC permissions" ```yaml apiVersion: v1 kind: ServiceAccount metadata: name: pod-delete-sa namespace: default labels: name: pod-delete-sa app.kubernetes.io/part-of: litmus apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: pod-delete-sa namespace: default labels: name: pod-delete-sa app.kubernetes.io/part-of: litmus rules: apiGroups: [""] resources: ["pods"] verbs: ["create","delete","get","list","patch","update", "deletecollection"] apiGroups: [""] resources: ["events"] verbs: ["create","get","list","patch","update"] apiGroups: [""] resources: ["configmaps"] verbs: ["get","list",] apiGroups: [""] resources: ["pods/log"] verbs: ["get","list","watch"] apiGroups: [""] resources: ["pods/exec"] verbs: ["get","list","create"] apiGroups: ["apps"] resources: ["deployments","statefulsets","replicasets", "daemonsets"] verbs: ["list","get"] apiGroups: ["apps.openshift.io"] resources: ["deploymentconfigs"] verbs: ["list","get"] apiGroups: [""] resources: ["replicationcontrollers"] verbs: ["get","list"] apiGroups: ["argoproj.io"] resources: ["rollouts"] verbs: ["list","get"] apiGroups: ["batch"] resources: ["jobs"] verbs: ["create","list","get","delete","deletecollection"] apiGroups: ["litmuschaos.io"] resources: ["chaosengines","chaosexperiments","chaosresults"] verbs: ["create","list","get","patch","update","delete"] apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: pod-delete-sa namespace: default labels: name: pod-delete-sa app.kubernetes.io/part-of: litmus roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: pod-delete-sa subjects: kind: ServiceAccount name: pod-delete-sa namespace: default ``` Use this sample RBAC manifest to create a chaosServiceAccount in the desired (app) namespace. This example consists of the minimum necessary role permissions to execute the

??? info "check the experiment tunables" <h2>Optional Fields</h2> <table> <tr> <th> Variables </th> <th> Description </th> <th> Notes </th> </tr> <tr> <td> TOTALCHAOSDURATION </td> <td> The time duration for chaos insertion (in sec) </td> <td> Defaults to 15s, <b>NOTE:</b> Overall run duration of the experiment may exceed the TOTALCHAOSDURATION by a few min </td> </tr> <tr> <td> CHAOS_INTERVAL </td> <td> Time interval b/w two successive pod failures (in sec) </td> <td> Defaults to 5s </td> </tr> <tr> <td> RANDOMNESS </td> <td> Introduces randomness to pod deletions with a minimum period defined by CHAOS_INTERVAL </td> <td> It supports true or false. Default value: false </td> </tr> <tr> <td> FORCE </td> <td> Application Pod deletion mode. <code>false</code> indicates graceful deletion with default termination period of 30s. <code>true</code> indicates an immediate forceful deletion with 0s grace period</td> <td> Default to <code>true</code>, With <code>terminationGracePeriodSeconds=0</code> </td> </tr> <tr> <td> TARGET_PODS </td> <td> Comma separated list of application pod name subjected to pod delete chaos</td> <td> If not provided, it will select target pods randomly based on provided appLabels</td> </tr> <tr> <td> PODSAFFECTEDPERC </td> <td> The Percentage of total pods to target </td> <td> Defaults to 0 (corresponds to 1 replica), provide numeric value only </td> </tr> <tr> <td> RAMP_TIME </td> <td> Period to wait before and after injection of chaos in sec </td> <td> </td> </tr> <tr> <td> SEQUENCE </td> <td> It defines sequence of chaos execution for multiple target pods </td> <td> Default value: parallel. Supported: serial, parallel </td> </tr> </table> Refer the and to tune the common tunables for all experiments and pod specific tunables. The targeted pod can be deleted `forcefully` or `gracefully`. It can be tuned with the `FORCE` env. It will delete the pod forcefully if `FORCE` is provided as `true` and it will delete the pod gracefully if `FORCE` is provided as `false`. Use the following example to tune this: ```yaml apiVersion: litmuschaos.io/v1alpha1 kind: ChaosEngine metadata: name: engine-nginx spec: engineState: "active" annotationCheck: "false" appinfo: appns: "default" applabel: "app=nginx" appkind: "deployment" chaosServiceAccount: pod-delete-sa experiments: name: pod-delete spec: components: env: name: FORCE value: 'true' name: TOTALCHAOSDURATION value: '60' ``` The multiple iterations of chaos can be tuned via setting `CHAOS_INTERVAL` ENV. Which defines the delay between each iteration of chaos. Use the following example to tune this: ```yaml apiVersion: litmuschaos.io/v1alpha1 kind: ChaosEngine metadata: name: engine-nginx spec: engineState: "active" annotationCheck: "false" appinfo: appns: "default" applabel: "app=nginx" appkind: "deployment" chaosServiceAccount: pod-delete-sa experiments: name: pod-delete spec: components: env: name: CHAOS_INTERVAL value: '15' name: TOTALCHAOSDURATION value: '60' ``` The randomness in the chaos interval can be enabled via setting `RANDOMNESS` ENV to `true`. It supports boolean values. The default value is `false`. The chaos interval can be tuned via `CHAOS_INTERVAL` ENV. If `CHAOS_INTERVAL` is set in the form of `l-r` i.e, `5-10` then it will select a random interval between l & r. If `CHAOS_INTERVAL` is set in the form of `value` i.e, `10` then it will select a random interval between 0 & value. Use the following example to tune this: ```yaml apiVersion: litmuschaos.io/v1alpha1 kind: ChaosEngine metadata: name: engine-nginx spec: engineState: "active" annotationCheck: "false" appinfo: appns: "default" applabel: "app=nginx" appkind: "deployment" chaosServiceAccount: pod-delete-sa experiments: name: pod-delete spec: components: env: name: RANDOMNESS value: 'true' name: TOTALCHAOSDURATION value: '60' name: CHAOS_INTERVAL value: '5-10' ```