Package: chaotic-ngine
Chaotic - Chaos for CloudsChaotic evaluates a plan, how it will bring chaos in your Cloud environment.Depending on the Cloud API used, it may kill allocations (Hashicorp Nomad), reboot or stop/start virtual machines in your Cloud environment.With no arguments given, Chaotic runs as a "one shot" meant to be executed as cron job. Passing--periodicruns it as daemon with configurable interval--interval 5in minutes (1 is the default).
NOTE: The config is re-read on every interval, no need to restart the service after changing the config.CloudsCurrently implemented Clouds:DigitalOceanVultrHetzner CloudProxmox KVMCloudStackHashicorp NomadExoscalecloudscale.chInstallpip3 install -U chaotic-ngineConfigureCreate a file namedconfig.yamlor use the env varCHAOTIC_CONFIGto point to a config file (also see the example directory):export CHAOTIC_CONFIG=config_nomad.yamlExclude timesDefine times when the bot should not doing real actions (it will run in dry-run):---kind:...excludes:weekdays:-Sun-Sattimes_of_day:-22:00-08:00-11:00-14:00days_of_year:-Jan01-Apr01-May01-Aug01-Dec24ExoscaleChaotic will stop a server selected by an optional filter tag and stop/start it with a delay of a configurable time (default 60s).export EXOSCALE_API_KEY="..."
export EXOSCALE_API_SECRET="..."---kind:exoscaledry_run:falseconfigs:# Optional, filter tagtag:key:chaosvalue:enabled# Optional, 60 seconds is the defaultwait_before_restart:60CloudStackChaotic will stop a server selected by an optional filter tag and stop/start it with a delay of a configurable time (default 60s).export CLOUDSTACK_API_KEY="..."
export CLOUDSTACK_API_SECRET="..."
export CLOUDSTACK_API_ENDPOINT="..."---kind:cloudstackdry_run:falseconfigs:# Optional, filter tagtag:key:chaosvalue:enabled# Optional, 60 seconds is the defaultwait_before_restart:60VultrChaotic will stop a server selected by an optional filter tag and stop/start it with a delay of a configurable time (default 60s).export VULTR_API_KEY="..."---kind:vultrdry_run:trueconfigs:# Optional instance tag filtertag:"chaos=opt-in"# Optional, 60 seconds is the defaultwait_before_restart:60Cloudscale.chChaotic will stop a server selected by an optional filter tag and stop/start it with a delay of a configurable time (default 60s).Configexport CLOUDSCALE_API_TOKEN="..."---kind:cloudscale_chdry_run:trueconfigs:# Optional server tag filterfilter_tag:"chaos=opt-in"# Optional, 60 seconds is the defaultwait_before_restart:60Hetzner CloudChaotic will stop a server selected by an optional filter label and stop/start it with a delay of a configurable time (default 60s).Configexport HCLOUD_API_TOKEN=...---kind:hclouddry_run:falseconfigs:# Optional server label filterlabel:"chaos=enabled"# Optional, 60 seconds is the defaultwait_before_restart:60DigitalOcean CloudChaotic will stop a droplet selected by an optional filter tag and stop/start it with a delay of a configurable time (default 60s).Configexport DIGITALOCEAN_ACCESS_TOKEN=...---kind:digitaloceandry_run:falseconfigs:# Optional droplet tag filtertag:"chaos:enabled"# Optional, 60 seconds is the defaultwait_before_restart:60Nomad JobChaotic will send an allocation signal to an allocation in the available namespaces selected by an allow list.Configexport NOMAD_ADDR=http://nomad.example.com:4646---kind:nomaddry_run:trueconfigs:experiments:-job# Signals to choose fromsignals:-SIGKILL# Optional: namespace allowlistnamespace_allowlist:-example-prod-foobar-prod# Optional: namespace denylistnamespace_denylist:-default# Optional: job type skip listjob_type_skiplist:-system-batch-sysbatch# Optional: job name skip listjob_skiplist:-my-job-name# Optional: Add a meta tag in your nomad job "chaotic" = False to opt-outjob_meta_opt_key:chaoticNomad NodeChaotic will drain a node and set it to be ineligible for some time.Configexport NOMAD_ADDR=http://nomad.example.com:4646---kind:nomaddry_run:trueconfigs:experiments:-node# Optional: Node drain deadline in seconds, default 10node_drain_deadline_seconds:15# Optional: Skip nodes in these classesnode_class_skiplist:-storage# Optional: Skip nodes with these namesnode_skiplist:-node1-node5# Optional: Wait for this amount of seconds before set node to be eligible again, default 60node_wait_for:100# Optional: Also drain system jobs, default falsenode_drain_system_jobs:true# Optional: Drain multiple nodes in one run in percent, fallback 1 nodenode_drain_amount_in_percent:30Proxmox KVMChaotic will stop a VM stop/start it with a delay of a configurable time (default 60s).export PROXMOX_API_HOST="pve1.example.com"
export PROXMOX_API_USER="root@pam"
export PROXMOX_API_PASSWORD="..."---kind:proxmox_kvmdry_run:falseconfigs:# Optional: Do not shutdown VMs having a lower uptime in minutesmin_uptime:60# Optional: Do not shutdown VMs in this name listdenylist:-my-single-vm# Optional: 60 seconds is the defaultwait_before_restart:60RunCLIchaos-ngineDockerOne shot:docker run -ti --rm -v $PWD/examples/config_nomad.yaml:/app/config.yaml -e TZ=Europe/Zurich -e NOMAD_ADDR=$NOMAD_ADDR --name chaotic ghcr.io/ngine-io/chaotic:latestAs service:docker run -ti --rm -v $PWD/examples/config_nomad.yaml:/app/config.yaml -e TZ=Europe/Zurich -e NOMAD_ADDR=$NOMAD_ADDR --name chaotic ghcr.io/ngine-io/chaotic:latest --periodicLogsWhat you should see (e.g. for kind cloudscale.ch):2021-06-09 09:01:25,433 - cloudscale.log:INFO:Started, version: 0.6.2
2021-06-09 09:01:25,433 - cloudscale.log:INFO:Using profile default
2021-06-09 09:01:25,433 - cloudscale.log:INFO:API Token used: xyz...
2021-06-09 09:01:25,433 - chatic:INFO:Querying with filter_tag: None
2021-06-09 09:01:25,433 - cloudscale.log:INFO:HTTP GET to https://api.cloudscale.ch/v1/servers
2021-06-09 09:01:25,651 - cloudscale.log:INFO:HTTP status code 200
2021-06-09 09:01:25,652 - chatic:INFO:Choose server app3
2021-06-09 09:01:25,653 - chatic:INFO:Stopping server app3
2021-06-09 09:01:25,653 - cloudscale.log:INFO:HTTP POST to https://api.cloudscale.ch/v1/servers/d5628484-a6eb-4ea9-b3ef-ba8da2bb9fe0/stop
2021-06-09 09:01:26,336 - cloudscale.log:INFO:HTTP status code 204
2021-06-09 09:01:26,336 - chatic:INFO:Sleeping for server 60
2021-06-09 09:02:26,393 - cloudscale.log:INFO:HTTP POST to https://api.cloudscale.ch/v1/servers/d5628484-a6eb-4ea9-b3ef-ba8da2bb9fe0/start
2021-06-09 09:02:26,955 - cloudscale.log:INFO:HTTP status code 204
2021-06-09 09:02:26,956 - chatic:INFO:done