Project: Chaos Mesh (Observability and Analysis / Chaos Engineering)
File: design.md

<!-- TOC --> - - - - - - - - - - <!-- /TOC --> WIP: This document is still under development; everything is not stable, might change frequently. As the RFC of Chaos Mesh Workflow is stable, we create this document as the reference for implementation. We will not define a specific struct here, but we present pseudo-code for interface and main logic. An instance of Chaos Mesh Workflow is basically like a tree: each action is described by a node; a root node represents for entry-point; nodes with certain types could have children nodes; depends on the type of this node, and children nodes execute serial or parallel; node with a certain type could select which children nodes to run(conditional branch); So Chaos Mesh Workflow is mostly like this: Argo It doesn't like: BPMN Do we really need a tree? How about a pipeline? Workflow: a resource that defines the orchestration of chaos experiments. Template: the node in Workflow "tree", represents for operation. There are various operations: like "create chaos experiments", "wait for 5 minutes", "create a serial job with children templates", etc. Node: a running/completed step within a workflow, it represents the status; We could think of it as an instance rendered from a Template. CronWorkflow: a way to execute workflow with a schedule. The relationship between CronWorkflow and Workflow is like what between and . Here are two kubernetes CRDs: `Workflow` and `WorkflowNode`, they are represents `Workflow` and `Node` in the core concepts. The spec in `Workflow` stores the `Template`s and entry of that workflow, status in `Workflow` store the reference of `WorkflowNode`s. Each `WorkflowNode` means a running or finished `Node`, its spec is generated by `Template`, as there are many type of `Template` and we only have one `WorkflowNode`, so not all fields in spec/status are used at once. We use `Condition` in `WorkflowNode` as the trigger of schedule operation: such like `Accomplished` and `DeadlineExceed`, all the `WorkflowNode` will be treated as finished by one of the conditions. When a `Workflow` is committed into kubernetes, a lot of `WorkflowNode` will be created with the processing of that workflow, they will make a tree and the root is the `Workflow`. That reference also represented by `OwnerReference`, so as `Workflow` deleted, all related `WorkflowNode` will be deleted cascaded by garbage collector. Why kubernetes pattern again? Finally, we back to the kubernetes pattern, which means declarative API, continuous operating until current states up to desired states. We have tried to use non-kubernetes pattern before, such as using webhook for injecting chaos, and two-phase controller. Finally these are not the best practices for declarative API, and should be modified again and

All the conditions only observed by the current state of the real world, they do not mean any historical events, and we could not make any decision by the changes of conditions. Here are two important conditions that affect the schedule of the workflow: `Accomplished` and `DeadlineExceed`. Both of these conditions mean that the current node is finished, and we should not make more operations. `Accomplished` usually be used with `Serial` and `Parallel` nodes, which means this node becomes accomplished because all of the children become finished. `Deadline` usually be used with the `Chaos` node which has a field `deadline` in their spec. Here is one condition called `ChaosInjected` on chaos node, it means the chaos CR object is created. WIP with other type of workflow node. We follow the kubernetes pattern, make operations based on desired states and current states. Each reconciler corresponds to one type of operations, it is might a little different with other controller, almost all the reconcilers make effects on `WorkflowNode`. Workflow entry reconciler watches on workflow, creates the first `WorkflowNode` referenced by entry, then updates the status of the workflow. Deadline reconciler will update the condition `DeadlineExceeded` when the deadline exceeded. We use three fields to describe current statue of serial node: ExpectedChildren *int ActiveChildren []corev1.LocalObjectReference FinishedChildren []corev1.LocalObjectReference Serial reconciler will create new `WorkflowNode` if `ActiveChildren` is empty, then add this object into `ActiveChildren`. If `len(FinishedChildren) == ExpectedChildren`, serial reconciler will update the `Accomplish` condition to true. Currently, `Serial`, `Parallel` and `Task` all have these fields mentioned above. Accomplish watcher keeps watching on all `WorkflowNode`s, as one of `Accomplished` and `DeadlineExceed` becomes true, it will update its parent's status. For supporting users could write their codes for judgment based on current workflow status, we provide a mechanism also called "Downward API". Likes Kubernetes Downward API, it also injects status as a file into the pod. The feature is enabled by default, there are no configurations for it currently. We will create a ConfigMap Resource, the content of it is Workflow CRD object, contains spec and status as JSON. When creating the Pod, it will be mounted on `/var/run/chaos-mesh/`, so the user could access this file like `cat /var/run/chaos-mesh/workflow.json`. The content of Downward API is a static snapshot about workflow object, it will NOT change when Workflow object in kubernetes updates. We want Chaos Mesh Workflow could drive these things at the same time: <= 100 workflows each workflow contains <= 100 templates "scheduling next operation" should be completed in 1 seconde This is only about the workflow, it doesn't consider performance about Chaos Experiments We are not using schema-less things now. Support DAG or not? No. I think it's not enough necessary. Assertion SDK / WebAPI for Task Split controller-manager and workflow-engine as two binary? Not for now. Should `Node` be a standalone CRD? Yes, it's more easier for control the desire states.