Project: Krkn (Observability and Analysis / Chaos Engineering)
File: README.md

This tool, designed for Redhat Kraken, operates through the command line and offers recommendations for chaos testing. It suggests probable chaos test cases that can disrupt application services by analyzing their behavior and assessing their susceptibility to specific fault types. This tool profiles an application and gathers telemetry data such as CPU, Memory, and Network usage, analyzing it to suggest probable chaos scenarios. For optimal results, it is recommended to activate the utility while the application is under load. Openshift Or Kubernetes Environment where the application is hosted Access to the metrics via the exposed Prometheus endpoint Python3.9 To run ``` $ python3.9 -m venv chaos $ source chaos/bin/activate $ git clone https://github.com/krkn-chaos/krkn.git $ cd krkn $ pip3 install -r requirements.txt Edit configuration file: $ vi config/recommender_config.yaml $ python3.9 utils/chaosrecommender/chaosrecommender.py -c utils/chaosrecommender/recommenderconfig.yaml ``` Follow the prompts to provide the required information. To run the recommender with a config file specify the config file path with the `-c` argument. You can customize the default values by editing the `recommender_config.yaml` file. The configuration file contains the following options: `application`: Specify the application name. `namespaces`: Specify the namespaces names (separated by coma or space). If you want to profile `labels`: Specify the labels (not used). `kubeconfig`: Specify the location of the kubeconfig file (not used). `prometheus_endpoint`: Specify the prometheus endpoint (must). `auth_token`: Auth token to connect to prometheus endpoint (must). `scrape_duration`: For how long data should be fetched, e.g., '1m' (must). `chaos_library`: "kraken" (currently it only supports kraken). `jsonoutputfile`: True or False (by default False). `jsonoutputfolder_path`: Specify folder path where output should be saved. If empty the default path is

`chaos_tests`: (for output purpose only do not change if not needed) `GENERAL`: list of general purpose tests available in Krkn `MEM`: list of memory related tests available in Krkn `NETWORK`: list of network related tests available in Krkn `CPU`: list of memory related tests available in Krkn `threshold`: Specify the threshold to use for comparison and identifying outliers `cpu_threshold`: Specify the cpu threshold to compare with the cpu limits set on the pods and identify outliers `mem_threshold`: Specify the memory threshold to compare with the memory limits set on the pods and identify outliers TIP: to collect prometheus endpoint and token from your OpenShift cluster you can run the following commands: ``` prometheus_url=$(kubectl get routes -n openshift-monitoring prometheus-k8s --no-headers | awk '{print $2}') token=$(oc whoami -t) token=$(kubectl create token -n openshift-monitoring prometheus-k8s --duration=6h || oc sa new-token -n openshift-monitoring prometheus-k8s) ``` You can also provide the input values through command-line arguments launching the recommender with `-o` option: ``` -o, --options Evaluate command line options -a APPLICATION, --application APPLICATION Kubernetes application name -n NAMESPACES, --namespaces NAMESPACE Kubernetes application namespaces separated by space -l LABELS, --labels LABELS Kubernetes application labels -p PROMETHEUSENDPOINT, --prometheus-endpoint PROMETHEUSENDPOINT Prometheus endpoint URI -k KUBECONFIG, --kubeconfig KUBECONFIG Kubeconfig path -t TOKEN, --token TOKEN Kubernetes authentication token -s SCRAPEDURATION, --scrape-duration SCRAPEDURATION Prometheus scrape duration -i LIBRARY, --library LIBRARY Chaos library -L LOGLEVEL, --log-level LOGLEVEL log level (DEBUG, INFO, WARNING, ERROR, CRITICAL -J [FOLDERPATH], --json-output-file [FOLDERPATH] Create output file, the path to the folder can be specified, if not specified the default folder is used. -M MEM [MEM ...], --MEM MEM [MEM ...] Memory related chaos tests (space separated list) -C CPU [CPU ...], --CPU CPU [CPU ...] CPU related chaos tests (space separated list) -N NETWORK [NETWORK ...], --NETWORK NETWORK [NETWORK ...] Network related chaos tests (space separated list) -G GENERIC [GENERIC ...], --GENERIC GENERIC [GENERIC ...] Memory related chaos tests (space separated list) --threshold THRESHOLD Threshold --cputhreshold CPUTHRESHOLD CPU threshold to compare with the cpu limits --memthreshold MEMTHRESHOLD Memory threshold to compare with the memory limits ``` If you provide the input values through command-line arguments, the corresponding config file inputs would be ignored. To run the recommender image please visit the [krkn-hub](https://github.com/krkn-chaos/krkn-hub for further infos. After obtaining telemetry data, sourced either locally or from Prometheus, the tool conducts a comprehensive data analysis to detect anomalies. Employing the Z-score method and heatmaps, it identifies outliers by evaluating CPU, memory, and network usage against established limits. Services with Z-scores surpassing a specified threshold are categorized as outliers. This categorization classifies services as network, CPU, or memory-sensitive, consequently leading to the recommendation of relevant test cases. You can customize the thresholds and options used for data analysis and identifying the outliers by setting the threshold, cputhreshold and memthreshold parameters in the config. `recommender_config.yaml`: The configuration file containing default values for application, namespace, labels, and kubeconfig. Happy Chaos!