Project: Litmus (Observability and Analysis / Chaos Engineering)
File: pod-http-latency.md

It injects http response latency on the service whose port is provided as `TARGETSERVICEPORT` by starting proxy server and then redirecting the traffic through the proxy server. It can test the application's resilience to lossy/flaky http responses. !!! tip "Scenario: Add latency to the HTTP request" ??? info "View the uses of the experiment" coming soon ??? info "Verify the prerequisites" Ensure that Kubernetes Version > 1.17 Ensure that the Litmus Chaos Operator is running by executing <code>kubectl get pods</code> in operator namespace (typically, <code>litmus</code>).If not, install from <a href="https://v1-docs.litmuschaos.io/docs/getstarted/#install-litmus">here</a> Ensure that the <code>pod-http-latency</code> experiment resource is available in the cluster by executing <code>kubectl get chaosexperiments</code> in the desired namespace. If not, install from <a href="https://hub.litmuschaos.io/api/chaos/master?file=faults/kubernetes/pod-http-latency/fault.yaml">here</a> ??? info "View the default validations" The application pods should be in running state before and after chaos injection. !!! tip "NOTE" If you are using this experiment as part of a litmus workflow scheduled constructed & executed from chaos-center, then you may be making use of the RBAC, which is pre installed in the cluster as part of the agent setup. ??? note "View the Minimal RBAC permissions" ```yaml apiVersion: v1 kind: ServiceAccount metadata: name: pod-http-latency-sa namespace: default labels: name: pod-http-latency-sa app.kubernetes.io/part-of: litmus apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: pod-http-latency-sa namespace: default labels: name: pod-http-latency-sa app.kubernetes.io/part-of: litmus rules: apiGroups: [""] resources: ["pods"] verbs: ["create","delete","get","list","patch","update", "deletecollection"] apiGroups: [""] resources: ["events"] verbs: ["create","get","list","patch","update"] apiGroups: [""] resources: ["configmaps"] verbs: ["get","list",] apiGroups: [""] resources: ["pods/log"] verbs: ["get","list","watch"] apiGroups: [""] resources: ["pods/exec"] verbs: ["get","list","create"] apiGroups: ["apps"] resources: ["deployments","statefulsets","replicasets", "daemonsets"] verbs: ["list","get"] apiGroups: ["apps.openshift.io"] resources: ["deploymentconfigs"] verbs: ["list","get"] apiGroups: [""] resources: ["replicationcontrollers"] verbs: ["get","list"] apiGroups: ["argoproj.io"] resources: ["rollouts"] verbs: ["list","get"] apiGroups: ["batch"] resources: ["jobs"] verbs: ["create","list","get","delete","deletecollection"] apiGroups: ["litmuschaos.io"] resources: ["chaosengines","chaosexperiments","chaosresults"] verbs: ["create","list","get","patch","update","delete"] apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: pod-http-latency-sa namespace: default labels: name: pod-http-latency-sa app.kubernetes.io/part-of: litmus roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: pod-http-latency-sa subjects: kind: ServiceAccount name: pod-http-latency-sa namespace: default ``` Use this sample RBAC manifest to create a chaosServiceAccount in the desired (app) namespace. This example consists of the minimum necessary role permissions to execute the experiment. ??? info "check the experiment tunables" <h2>Mandatory Fields</h2> <table> <tr> <th> Variables </th> <th> Description </th> <th> Notes </th> </tr> <tr> <td> TARGETSERVICEPORT </td> <td> Port of the service to target</td> <td>Defaults to port 80 </td> </tr> <tr> <td> LATENCY </td> <td> Latency value in ms to be added to requests</td> <td> Defaults to 2000 </td> </tr> </table> <h2>Optional Fields</h2> <table> <tr> <th> Variables </th> <th> Description </th> <th> Notes </th> </tr> <tr> <td> PROXY_PORT </td> <td> Port where the proxy will be listening for requests</td> <td> Defaults to 20000 </td> </tr> <tr> <td> NETWORK_INTERFACE </td> <td> Network interface to be used for the proxy</td> <td> Defaults to `eth0` </td> </tr> <tr> <td> TOXICITY </td> <td> Percentage of HTTP requests to be affected </td> <td> Defaults to 100 </td> </tr> <tr> <td> CONTAINER_RUNTIME </td> <td> container runtime interface for the cluster</td> <td> Defaults to containerd, supported values: docker, containerd and crio for litmus and only docker for pumba LIB </td> </tr> <tr> <td> SOCKET_PATH </td> <td> Path of the containerd/crio/docker socket file </td> <td> Defaults to

</td> </tr> <tr> <td> TOTALCHAOSDURATION </td> <td> The duration of chaos injection (seconds) </td> <td> Default (60s) </td> </tr> <tr> <td> TARGET_PODS </td> <td> Comma separated list of application pod name subjected to pod http latency chaos</td> <td> If not provided, it will select target pods randomly based on provided appLabels</td> </tr> <tr> <td> PODSAFFECTEDPERC </td> <td> The Percentage of total pods to target </td> <td> Defaults to 0 (corresponds to 1 replica), provide numeric value only </td> </tr> <tr> <td> LIB_IMAGE </td> <td> Image used to run the netem command </td> <td> Defaults to `litmuschaos/go-runner:latest` </td> </tr> <tr> <td> RAMP_TIME </td> <td> Period to wait before and after injection of chaos in sec </td> <td> </td> </tr> <tr> <td> SEQUENCE </td> <td> It defines sequence of chaos execution for multiple target pods </td> <td> Default value: parallel. Supported: serial, parallel </td> </tr> </table> Refer the and to tune the common tunables for all experiments and pod specific tunables. It defines the port of the targeted service that is being targeted. It can be tuned via `TARGETSERVICEPORT` ENV. Use the following example to tune this: ```yaml apiVersion: litmuschaos.io/v1alpha1 kind: ChaosEngine metadata: name: engine-nginx spec: engineState: "active" annotationCheck: "false" appinfo: appns: "default" applabel: "app=nginx" appkind: "deployment" chaosServiceAccount: pod-http-latency-sa experiments: name: pod-http-latency spec: components: env: name: TARGETSERVICEPORT value: "80" ``` It defines the port on which the proxy server will listen for requests. It can be tuned via `PROXY_PORT` ENV. Use the following example to tune this: ```yaml apiVersion: litmuschaos.io/v1alpha1 kind: ChaosEngine metadata: name: engine-nginx spec: engineState: "active" annotationCheck: "false" appinfo: appns: "default" applabel: "app=nginx" appkind: "deployment" chaosServiceAccount: pod-http-latency-sa experiments: name: pod-http-latency spec: components: env: name: PROXY_PORT value: '8080' name: TARGETSERVICEPORT value: "80" ``` It defines the latency value to be added to the http request. It can be tuned via `LATENCY` ENV. Use the following example to tune this: ```yaml apiVersion: litmuschaos.io/v1alpha1 kind: ChaosEngine metadata: name: engine-nginx spec: engineState: "active" annotationCheck: "false" appinfo: appns: "default" applabel: "app=nginx" appkind: "deployment" chaosServiceAccount: pod-http-latency-sa experiments: name: pod-http-latency spec: components: env: name: LATENCY value: '2000' name: TARGETSERVICEPORT value: "80" ``` It defines the toxicity value to be added to the http request. It can be tuned via `TOXICITY` ENV. Toxicity value defines the percentage of the total number of http requests to be affected. Use the following example to tune this: ```yaml apiVersion: litmuschaos.io/v1alpha1 kind: ChaosEngine metadata: name: engine-nginx spec: engineState: "active" annotationCheck: "false" appinfo: appns: "default" applabel: "app=nginx" appkind: "deployment" chaosServiceAccount: pod-http-latency-sa experiments: name: pod-http-latency spec: components: env: name: TOXICITY value: "100" name: TARGETSERVICEPORT value: "80" ``` It defines the network interface to be used for the proxy. It can be tuned via `NETWORK_INTERFACE` ENV. Use the following example to tune this: ```yaml apiVersion: litmuschaos.io/v1alpha1 kind: ChaosEngine metadata: name: engine-nginx spec: engineState: "active" annotationCheck: "false" appinfo: appns: "default" applabel: "app=nginx" appkind: "deployment" chaosServiceAccount: pod-http-latency-sa experiments: name: pod-http-latency spec: components: env: name: NETWORK_INTERFACE value: "eth0" name: TARGETSERVICEPORT value: '80' ``` It defines the `CONTAINERRUNTIME` and `SOCKETPATH` ENV to set the container runtime and socket file path. `CONTAINER_RUNTIME`: It supports `docker`, `containerd`, and `crio` runtimes. The default value is `docker`. `SOCKET_PATH`: It contains path of docker socket file by default(`/run/containerd/containerd.sock`). For other runtimes provide the appropriate path. Use the following example to tune this: ```yaml apiVersion: litmuschaos.io/v1alpha1 kind: ChaosEngine metadata: name: engine-nginx spec: engineState: "active" annotationCheck: "false" appinfo: appns: "default" applabel: "app=nginx" appkind: "deployment" chaosServiceAccount: pod-http-latency-sa experiments: name: pod-http-latency spec: components: env: name: CONTAINER_RUNTIME value: 'containerd' name: SOCKET_PATH value: '/run/containerd/containerd.sock' name: TARGETSERVICEPORT value: "80" ```