Project: Litmus (Observability and Analysis / Chaos Engineering)
File: disk-fill.md

It causes Disk Stress by filling up the ephemeral storage of the pod on any given node. It causes the application pod to get evicted if the capacity filled exceeds the pod's ephemeral storage limit. It tests the Ephemeral Storage Limits, to ensure those parameters are sufficient. It tests the application's resiliency to disk stress/replica evictions. !!! tip "Scenario: Fill ephemeral-storage" ??? info "View the uses of the experiment" coming soon ??? info "Verify the prerequisites" Ensure that Kubernetes Version > 1.16 Ensure that the Litmus Chaos Operator is running by executing <code>kubectl get pods</code> in operator namespace (typically, <code>litmus</code>).If not, install from <a href="https://v1-docs.litmuschaos.io/docs/getstarted/#install-litmus">here</a> Ensure that the <code>disk-fill</code> experiment resource is available in the cluster by executing <code>kubectl get chaosexperiments</code> in the desired namespace. If not, install from <a href="https://hub.litmuschaos.io/api/chaos/master?file=faults/kubernetes/disk-fill/fault.yaml">here</a> Appropriate Ephemeral Storage Requests and Limits should be set for the application before running the experiment. An example specification is shown below: ```yaml apiVersion: v1 kind: Pod metadata: name: frontend spec: containers: name: db image: mysql env: name: MYSQLROOTPASSWORD value: "password" resources: requests: ephemeral-storage: "2Gi" limits: ephemeral-storage: "4Gi" name: wp image: wordpress resources: requests: ephemeral-storage: "2Gi" limits: ephemeral-storage: "4Gi" ``` ??? info "View the default validations" The application pods should be in running state before and after chaos injection. !!! tip "NOTE" If you are using this experiment as part of a litmus workflow scheduled constructed & executed from chaos-center, then you may be making use of the RBAC, which is pre installed in the cluster as part of the agent setup. ??? note "View the Minimal RBAC permissions" ```yaml apiVersion: v1 kind: ServiceAccount metadata: name: disk-fill-sa namespace: default labels: name: disk-fill-sa app.kubernetes.io/part-of: litmus apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: disk-fill-sa namespace: default labels: name: disk-fill-sa app.kubernetes.io/part-of: litmus rules: apiGroups: [""] resources: ["pods"] verbs: ["create","delete","get","list","patch","update", "deletecollection"] apiGroups: [""] resources: ["events"] verbs: ["create","get","list","patch","update"] apiGroups: [""] resources: ["configmaps"] verbs: ["get","list",] apiGroups: [""] resources: ["pods/log"] verbs: ["get","list","watch"] apiGroups: [""] resources: ["pods/exec"] verbs: ["get","list","create"] apiGroups: ["apps"] resources: ["deployments","statefulsets","replicasets", "daemonsets"] verbs: ["list","get"] apiGroups: ["apps.openshift.io"] resources: ["deploymentconfigs"] verbs: ["list","get"] apiGroups: [""] resources: ["replicationcontrollers"] verbs: ["get","list"] apiGroups: ["argoproj.io"] resources: ["rollouts"] verbs: ["list","get"] apiGroups: ["batch"] resources: ["jobs"] verbs: ["create","list","get","delete","deletecollection"] apiGroups: ["litmuschaos.io"] resources: ["chaosengines","chaosexperiments","chaosresults"] verbs: ["create","list","get","patch","update","delete"] apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: disk-fill-sa namespace: default labels: name: disk-fill-sa app.kubernetes.io/part-of: litmus roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: disk-fill-sa subjects: kind: ServiceAccount name: disk-fill-sa namespace: default ``` Use this sample RBAC manifest to create a chaosServiceAccount in the desired (app) namespace. This example consists of the minimum necessary role permissions to execute the experiment. ??? info "check the experiment tunables" <h2>Mandatory Fields</h2> <table> <tr> <th> Variables </th> <th> Description </th> <th> Notes </th> </tr> <tr> <td> FILL_PERCENTAGE </td> <td> Percentage to fill the Ephemeral storage limit </td> <td> Can be set to more than 100 also, to force evict the pod. The ephemeral-storage limits must be set in targeted pod to use this ENV.</td> </tr> <tr> <td> EPHEMERALSTORAGEMEBIBYTES </td> <td> Ephemeral storage which need to fill (unit: MiBi)</td> <td>It is mutually exclusive with the <code>FILLPERCENTAGE</code>

If both are provided then it will use the <code>FILLPERCENTAGE</code></td> </tr> </table> <h2>Optional Fields</h2> <table> <tr> <th> Variables </th> <th> Description </th> <th> Notes </th> </tr> <tr> <td> TARGET_CONTAINER </td> <td> Name of container which is subjected to disk-fill </td> <td> If not provided, the first container in the targeted pod will be subject to chaos </td> </tr> <tr> <td> CONTAINER_RUNTIME </td> <td> container runtime interface for the cluster</td> <td> Defaults to containerd, supported values: docker, containerd and crio for litmus </td> </tr> <tr> <td> SOCKET_PATH </td> <td> Path of the containerd/crio/docker socket file </td> <td> Defaults to <code>/run/containerd/containerd.sock</code> </td> </tr> <tr> <td> TOTALCHAOSDURATION </td> <td> The time duration for chaos insertion (sec) </td> <td> Defaults to 60s </td> </tr> <tr> <td> TARGET_PODS </td> <td> Comma separated list of application pod name subjected to disk fill chaos</td> <td> If not provided, it will select target pods randomly based on provided appLabels</td> </tr> <tr> <td> DATABLOCKSIZE </td> <td> It contains data block size used to fill the disk(in KB)</td> <td> Defaults to 256, it supports unit as KB only</td> </tr> <tr> <td> PODSAFFECTEDPERC </td> <td> The Percentage of total pods to target </td> <td> Defaults to 0 (corresponds to 1 replica), provide numeric value only </td> </tr> <tr> <td> LIB </td> <td> The chaos lib used to inject the chaos </td> <td> Defaults to `litmus` supported litmus only </td> </tr> <tr> <td> LIB_IMAGE </td> <td> The image used to fill the disk </td> <td> Defaults to <code>litmuschaos/go-runner:latest</code> </td> </tr> <tr> <td> RAMP_TIME </td> <td> Period to wait before injection of chaos in sec </td> <td> </td> </tr> <tr> <td> SEQUENCE </td> <td> It defines sequence of chaos execution for multiple target pods </td> <td> Default value: parallel. Supported: serial, parallel </td> </tr> </table> Refer the and to tune the common tunables for all experiments and pod specific tunables. It fills the `FILL_PERCENTAGE` percentage of the ephemeral-storage limit specified at `resource.limits.ephemeral-storage` inside the target application. Use the following example to tune this: ```yaml apiVersion: litmuschaos.io/v1alpha1 kind: ChaosEngine metadata: name: engine-nginx spec: engineState: "active" annotationCheck: "false" appinfo: appns: "default" applabel: "app=nginx" appkind: "deployment" chaosServiceAccount: disk-fill-sa experiments: name: disk-fill spec: components: env: name: FILL_PERCENTAGE value: '80' # in percentage name: TOTALCHAOSDURATION value: '60' ``` It fills the `EPHEMERALSTORAGEMEBIBYTES` MiBi of ephemeral storage of the targeted pod. It is mutually exclusive with the `FILLPERCENTAGE` ENV. If `FILLPERCENTAGE` ENV is set then it will use the percentage for the fill otherwise, it will fill the ephemeral storage based on `EPHEMERALSTORAGEMEBIBYTES` ENV. Use the following example to tune this: ```yaml apiVersion: litmuschaos.io/v1alpha1 kind: ChaosEngine metadata: name: engine-nginx spec: engineState: "active" annotationCheck: "false" appinfo: appns: "default" applabel: "app=nginx" appkind: "deployment" chaosServiceAccount: disk-fill-sa experiments: name: disk-fill spec: components: env: name: EPHEMERALSTORAGEMEBIBYTES value: '256' #in MiBi name: TOTALCHAOSDURATION value: '60' ``` It defines the size of the data block used to fill the ephemeral storage of the targeted pod. It can be tuned via `DATABLOCKSIZE` ENV. Its unit is `KB`. The default value of `DATABLOCKSIZE` is `256`. Use the following example to tune this: ```yaml apiVersion: litmuschaos.io/v1alpha1 kind: ChaosEngine metadata: name: engine-nginx spec: engineState: "active" annotationCheck: "false" appinfo: appns: "default" applabel: "app=nginx" appkind: "deployment" chaosServiceAccount: disk-fill-sa experiments: name: disk-fill spec: components: env: name: DATABLOCKSIZE value: '256' #in KB name: TOTALCHAOSDURATION value: '60' ``` It defines the `CONTAINERRUNTIME` and `SOCKETPATH` ENV to set the container runtime and socket file path. `CONTAINER_RUNTIME`: It supports `docker`, `containerd`, and `crio` runtimes. The default value is `containerd`. `SOCKET_PATH`: It contains path of containerd socket file by default(`/run/containerd/containerd.sock`). For other runtimes provide the appropriate path. Use the following example to tune this: ```yaml apiVersion: litmuschaos.io/v1alpha1 kind: