Skill: panlm/aws-fis-experiment-prepare (version 1.0.0)

--- name: aws-fis-experiment-prepare description: > Use when the user wants to prepare, create, or generate an AWS FIS (Fault Injection Service) experiment configuration. Triggers on "prepare FIS experiment", "create FIS experiment for [scenario]", "generate chaos experiment config", "准备 FIS 实验", "生成 [scenario] 混沌实验配置", "create experiment template for AZ power interruption", "set up fault injection test". Covers Scenario Library pre-built scenarios (AZ Power Interruption, AZ Application Slowdown, Cross-AZ Traffic Slowdown, Cross-Region Connectivity), custom single FIS actions (aws:rds:failover-db-cluster, aws:ec2:stop-instances, etc.), and SSM Automation-based fault injection for Amazon MSK (broker reboot) and ElastiCache Redis/Valkey (primary node reboot, replication group failover). --- # AWS FIS Experiment Prepare Generate all configuration files needed to run an AWS FIS experiment, then deploy via CloudFormation with self-healing iteration until the stack succeeds. Outputs a self-contained directory with a validated, deployed experiment template ready for execution. **Core principle:** Validate resource-action compatibility before generating files. Never deliver untested configuration — deploy and self-heal first. ## References **Always load for every experiment:** - `references/output-format.md` — directory layout, slug naming, README template - `references/cfn-base-template.md` — CFN skeleton (Parameters, IAM Role, Dashboard, FIS Template, Outputs) - `references/slug-conventions.md` — scenario/context slug abbreviations, resource naming, name length budget **Load conditionally by scenario:** - `references/az-power-interruption-guide.md` — AZ Power Interruption (sub-action pruning, tagging strategy, permissions) - `references/eks-pod-action-guide.md` — any `aws:eks:pod-*` action (RBAC Lambda, EKS Access Entry, Pod memory stress calculation) - `references/elasticache-redis-guide.md` — ElastiCache Redis/Valkey (native AZ power interruption, primary node reboot via SSM Automation, or replication group failover via SSM Automation) - `references/msk-guide.md` — Amazon MSK (broker reboot via SSM Automation — no native FIS action exists) **Utility scripts (execute, do not read as reference):** - `scripts/precheck-cfn-permissions.sh` — detects required CFN service role - `scripts/deploy-with-retry.sh` — validate + deploy + delete-on-fail - `scripts/rename-output-dir.sh` — appends FIS template ID to directory name **Script invocation:** `${SKILL_DIR}` refers to the absolute path of this skill's directory (where SKILL.md lives). Resolve it from the skill's filesystem location before running any scripts. ## Output Language Rule Detect the user's conversation language and use the **same language** for all output files (README.md, comments in JSON/YAML). - Chinese input → Chinese output - English input → English output - Mixed → follow the dominant language ## Prerequisites Required tools: - **AWS CLI** — `aws fis list-actions`, resource discovery, CloudFormation - **aws___search_documentation** / **aws___read_documentation** — FIS docs research - **jq** — required by `scripts/deploy-with-retry.sh` and `scripts/precheck-cfn-permissions.sh` **EKS Pod fault injection:** Cluster auth mode must be `API_AND_CONFIG_MAP` or `API`. Check: ```bash aws eks describe-cluster --name {CLUSTER} \ --query 'cluster.accessConfig.authenticationMode' ``` If `CONFIG_MAP` only, the user must update the cluster first. **MANDATORY:** For any `aws:eks:pod-*` action, follow `references/eks-pod-action-guide.md`. ## Workflow ### Step 1: Identify Scenario and Region **Classify user intent into one of these branches:** | Branch | Trigger | Additional Reference | |---|---|---| | Scenario Library | AZ Power Interruption, AZ App Slowdown, Cross-AZ/Region scenarios | Read AWS doc URL (table below) | | Custom FIS action | User specifies an action ID or describes a single fault | — | | Custom FIS action (ElastiCache) | ElastiCache AZ power interruption or Redis/Valkey failover | `references/elasticache-redis-guide.md` | | SSM Automation | Target service has no native FIS action (MSK, ElastiCache primary reboot, ElastiCache failover) | `references/msk-guide.md` or `references/elasticache-redis-guide.md` | If ambiguous, ask the user. **Scenario Library documentation URLs** (JSON templates are NOT available via CLI/API — read the doc to extract): | Scenario | Documentation URL | |---|---| | AZ Power Interruption | `https://docs.aws.amazon.com/en_us/fis/latest/userguide/az-availability-scenario.html` | | AZ Application Slowdown | `https://docs.aws.amazon.com/en_us/fis/latest/userguide/az-application-slowdown-scenario.html` | | Cross-AZ Traffic Slowdown | `https://docs.aws.amazon.com/en_us/fis/latest/userguide/cross-az-traffic-slowdown-scenario.html` | | Cross-Region Connectivity | `https://docs.aws.amazon.com/en_us/fis/latest/userguide/cross-region-scenario.html` | **Region detection order:** 1. User explicitly specifies 2. Infer from context (ARNs, previous conversation) 3. `aws configure get region` 4. Ask the user Store as `TARGET_REGION`. **Default experiment duration: `PT10M` (10 minutes)** for all scenarios and sub-actions unless the user specifies otherwise. For AZ Power Interruption, scale ARC Zonal Autoshift timing proportionally (ARC starts at minute 2, runs for 8 minutes at PT10M; formula: `startAfter = duration × (5/30)`). ### Step 2: Discover Target Resources #### For Scenario Library Scenarios **CRITICAL: Scenario Library experiment templates CANNOT be generated via FIS API.** You MUST call `aws___read_documentation` with the scenario URL (Step 1 table) to extract the JSON experiment template before generating any files. The documentation is the only authoritative source. **Target identification — prefer `resourceArns` over `resourceTags`:** - Use `resourceArns` (exact ARNs) for most resource types — more precise, no pre-tagging needed - Exception — these types do NOT support `resourceArns`, use `resourceTags` instead: - `aws:elasticache:replicationgroup` - `aws:ec2:autoscaling-group` - EKS pod actions use Kubernetes namespace + pod labels (neither `resourceArns` nor `resourceTags`) **`resourceArns` and `filters` are mutually exclusive.** FIS rejects targets that specify both. For AZ-scoped targeting, either use `resourceArns` with only the target AZ's ARNs, or use `resourceTags` + `filters` together. **If scenario is AZ Power Interruption:** follow `references/az-power-interruption-guide.md` for sub-action pruning, tagging strategy, permissions, and one-Stack-per-AZ design. **Ask the user:** 1. Which AZ to target (for AZ-level scenarios) 2. Which services to include (for AZ Power Interruption) — if user mentions specific services, include ONLY those + mandatory infrastructure sub-actions 3. Target resource identifiers (cluster IDs, instance IDs, etc.) #### For Custom FIS Actions ```bash aws fis get-action --id "ACTION_ID" --region TARGET_REGION ``` Extract required `targets` and `parameters`. Resolve user-provided identifiers to ARNs via AWS CLI. #### For Services Without Native FIS Actions (SSM Automation) 1. Confirm no native action exists: ```bash aws fis list-actions \ --query "actions[?starts_with(id, 'aws:{SERVICE}:')]" \ --region TARGET_REGION ``` 2. If empty, follow the service-specific guide: - Amazon MSK → `references/msk-guide.md` - ElastiCache primary node reboot → `references/elasticache-redis-guide.md` (Scenario 2) - Other services → not yet documented. Stop and inform the user. **Special case — ElastiCache:** Has a native FIS action for AZ-level impact (`aws:elasticache:replicationgroup-interrupt-az-power`) but **no native action for single-node reboot or replication group failover**. For primary node reboot, use SSM Automation per `references/elasticache-redis-guide.md` → Scenario 2. For replication group failover (TestFailover), use SSM Automation per `references/elasticache-redis-guide.md` → Scenario 3. 3. Discover resources via the target service's CLI (`aws kafka list-clusters`, etc.). ### Step 2.5: EKS Pod Action Setup Gate **If the experiment includes ANY `aws:eks:pod-*` action, complete this gate BEFORE Step 3.** Applicable actions: `aws:eks:pod-cpu-stress`, `aws:eks:pod-delete`, `aws:eks:pod-io-stress`, `aws:eks:pod-memory-stress`, `aws:eks:pod-network-blackhole-port`, `aws:eks:pod-network-latency`, `aws:eks:pod-network-packet-loss`. 1. Read the official documentation: ``` aws___read_documentation: url: https://docs.aws.amazon.com/fis/latest/userguide/eks-pod-actions.html ``` 2. Follow ALL requirements in `references/eks-pod-action-guide.md`: - Lambda-backed CFN Custom Resource for K8s RBAC (fixed names: `fis-sa`, `fis-experiment-role`, `fis-experiment-role-binding`) - EKS Access Entry for FIS Experiment Role (`Username: fis-experiment`) - Cluster auth mode check (`API_AND_CONFIG_MAP` or `API`) - Pod `readOnlyRootFilesystem: false` check - Network action limitations (no Fargate, no bridge mode) - **Pod memory stress threshold calculation** (if action is `aws:eks:pod-memory-stress`) — user's percent is total target, not injection value Do NOT skip. EKS pod actions have complex setup requirements that differ significantly from other FIS actions. ### Step 3: Validate Resource-Action Compatibility **CRITICAL GATE.** Before generating any files, verify that the user's actual resources are compatible with the chosen FIS action(s). #### 3a. Inspect the Actual Resource | User Says | CLI Command | Key Fields | |---|---|---| | RDS database | `aws rds describe-db-instances --db-instance-identifier {ID}` | `Engine`, `DBClusterIdentifier` | | RDS/Aurora cluster | `aws rds describe-db-clusters --db-cluster-identifier {ID}` | `Engine`, `EngineMode`, `MultiAZ` | | EC2 instance | `aws ec2 describe-instances --instance-ids {ID}` | `InstanceType`, `Placement.AvailabilityZone` | | EKS cluster | `aws eks describe-cluster --name {NAME}` | `accessConfig.authenticationMode`, `version` | | ElastiCache | `aws elasticache describe-replication-groups --replication-group-id {ID}` | `NodeGroupConfiguration`, `MultiAZ` | | ASG | `aws autoscaling describe-auto-scaling-groups --auto-scaling-group-names {NAME}` | `AvailabilityZones`, `Instances` | #### 3b. Cross-Check Against FIS Action Requirements ```bash aws fis get-action --id "ACTION_ID" --region TARGET_REGION \ --query 'action.targets' --output json ``` **Common incompatibility traps:** | FIS Action | Required resourceType | Incompatible With | Detection | |---|---|---|---| | `aws:rds:failover-db-cluster` | `aws:rds:cluster` | Standalone RDS (non-Aurora) | `DBClusterIdentifier` is null | | `aws:rds:reboot-db-instances` | `aws:rds:db` | Aurora clusters | `Engine` starts with `aurora` | | `aws:elasticache:replicationgroup-interrupt-az-power` | `aws:elasticache:replicationgroup` | Standalone ElastiCache nodes | No replication group | | `aws:ec2:stop-instances` | `aws:ec2:instance` | Spot instances | `InstanceLifecycle` = `spot` | #### 3c. Decision Gate - **Compatible** → proceed to Step 4. - **Incompatible** → explain the mismatch, suggest alternatives based on the actual resource type, ask the user to confirm or abort. Example alternatives: - Standalone RDS Multi-AZ → `aws:rds:reboot-db-instances` with `--force-failover` - Aurora cluster → `aws:rds:failover-db-cluster` - ElastiCache standalone → explain replication group is required #### 3d. For Scenario Library Scenarios Validate EACH included sub-action against its target resources. Only validate sub-actions that remain after service-scoped pruning (Step 2). ### Step 4: Determine Monitoring Configuration **Stop Conditions — default: `source: "none"` (no alarm).** Only create a CloudWatch Alarm if the user explicitly provides one. **Dashboard Metrics — comprehensive, per-service.** Group widgets by service, 3 widgets per service (availability, performance, errors/latency). Include only services actually affected by the experiment. | Service | Metrics | |---|---| | EC2 | `StatusCheckFailed`, `CPUUtilization`, `NetworkIn/Out`, `NetworkPacketsIn/Out` | | RDS/Aurora | `DatabaseConnections`, `ReadLatency`, `WriteLatency`, `AuroraReplicaLag`, `FreeableMemory` | | EKS | `pod_number_of_running_pods`, [REDACTED_SECRET], `node_cpu_utilization`, `node_memory_utilization` | | ElastiCache | `ReplicationLag`, `EngineCPUUtilization`, `CurrConnections`, `CacheHitRate`, `Evictions`, `IsMaster` | | ALB | `HealthyHostCount`, `UnHealthyHostCount`, `HTTPCode_ELB_5XX_Count`, `TargetResponseTime` | | NLB | `ActiveFlowCount`, `TCP_Client_Reset_Count`, `TCP_Target_Reset_Count` | ### Step 5: Generate Configuration Files **Create output directory:** ```bash # ─── Fill in from user's request + references/slug-conventions.md ─── SCENARIO_SLUG="..." # e.g., pod-delete, az-power-int, rds-failover TARGET_RESOURCE_ID="..." # e.g., my-aurora-cluster, i-0abc123def CONTEXT_NAME="" # optional (e.g., redis, msk); leave empty if N/A # ──────────────────────────────────────────────────────────────────── # Derived values (do not edit): TARGET_SLUG=$(echo "${TARGET_RESOURCE_ID}" | tr '[:upper:]' '[:lower:]' | tr ' :/' '-' | cut -c1-20) CONTEXT_SLUG=$(echo "${CONTEXT_NAME}" | tr '[:upper:]' '[:lower:]' | tr ' :/' '-' | cut -c1-10) TIMESTAMP=$(TZ=Asia/Shanghai date +%Y-%m-%d-%H-%M-%S) if [ -n "${CONTEXT_SLUG}" ]; then OUTPUT_DIR="./${TIMESTAMP}-${SCENARIO_SLUG}-${TARGET_SLUG}-${CONTEXT_SLUG}" else OUTPUT_DIR="./${TIMESTAMP}-${SCENARIO_SLUG}-${TARGET_SLUG}" fi mkdir -p "${OUTPUT_DIR}" ``` **REQUIRED:** Before generating `cfn-template.yaml`, read the `AWS::FIS::ExperimentTemplate` CloudFormation resource documentation: ``` aws___read_documentation: url: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-fis-experimenttemplate.html ``` **ALSO REQUIRED:** Search for CloudFormation examples for the resources used: ``` aws___search_documentation: search_phrase: "<CFN resource types in this experiment>" topics: ["cloudformation"] ``` **Generate files:** 1. **cfn-template.yaml** — use `references/cfn-base-template.md` as the skeleton. Extend with scenario-specific resources per: - `references/az-power-interruption-guide.md` (if AZ Power Interruption) - `references/eks-pod-action-guide.md` (if EKS pod actions) - `references/msk-guide.md` (if MSK) - `references/elasticache-redis-guide.md` (if ElastiCache) 2. **README.md** — use the template in `references/output-format.md`. ### Step 5.5: CFN Permission Pre-Check Run the precheck script to detect whether a CFN service role is required: ```bash CFN_ROLE_ARN=$("${SKILL_DIR}/scripts/precheck-cfn-permissions.sh") ``` If the caller lacks CloudFormation permissions, the script exits 1 with guidance — **stop and inform the user**. Otherwise, `CFN_ROLE_ARN` is either empty (no service role needed) or contains the required role ARN. ### Step 6: Deploy CFN Template (Self-Healing Loop) **Generate deployment parameters:** ```bash # See references/slug-conventions.md for the ExperimentName composition rule RANDOM_SUFFIX=$(LC_ALL=C tr -dc 'a-z0-9' < /dev/urandom | head -c6) if [ -n "${CONTEXT_SLUG}" ]; then EXPERIMENT_NAME="${SCENARIO_SLUG}-${TARGET_SLUG}-${CONTEXT_SLUG}-${RANDOM_SUFFIX}" else EXPERIMENT_NAME="${SCENARIO_SLUG}-${TARGET_SLUG}-${RANDOM_SUFFIX}" fi STACK_NAME="fis-${EXPERIMENT_NAME}" ``` **Deploy with self-healing retry loop** (maximum 5 attempts driven by the agent). The `deploy-with-retry.sh` script performs **one attempt** — the agent drives the loop externally. On each attempt: 1. Run `scripts/deploy-with-retry.sh`: ```bash "${SKILL_DIR}/scripts/deploy-with-retry.sh" \ "${OUTPUT_DIR}/cfn-template.yaml" \ "${STACK_NAME}" \ "${TARGET_REGION}" \ "${CFN_ROLE_ARN}" \ "ExperimentName=${EXPERIMENT_NAME}" \ "RandomSuffix=${RANDOM_SUFFIX}" ``` 2. Exit 0 → deployment succeeded, proceed to "On Successful Deployment". 3. Exit 1 (validation failed) or 2 (deployment failed, stack deleted) → analyze stderr output, fix `cfn-template.yaml`, increment attempt counter, re-invoke the script. 4. After 5 failed attempts → stop and report to the user with the last error, all fixes attempted, and the current `cfn-template.yaml`. **Common CFN errors and fixes:** | Error Pattern | Root Cause | Fix | |---|---|---| | `Property validation failure` | Invalid CFN property name/value | Fix the resource property | | `Template format error` | YAML syntax issue | Fix indentation/structure | | `Resource type not supported` | Resource unavailable in region | Check regional availability | | `Circular dependency` | Resources reference each other | Use `DependsOn` or restructure | | `RoleArn ... is invalid` | IAM role not yet propagated | Add `DependsOn` for IAM role | | Empty `logConfiguration` | AZ Power Interruption doc artifact | Remove the `logConfiguration` block | #### On Successful Deployment 1. Extract stack outputs: ```bash aws cloudformation describe-stacks \ --stack-name "${STACK_NAME}" \ --query 'Stacks[0].Outputs' \ --region "${TARGET_REGION}" --output table ``` 2. Update `README.md` with actual stack name, template ID, dashboard URL, and cleanup command. Replace ALL `{STACK_NAME}` placeholders — do NOT leave placeholders in the final output. ### Step 7: Rename Output Directory with Template ID Run the rename script: ```bash NEW_OUTPUT_DIR=$("${SKILL_DIR}/scripts/rename-output-dir.sh" \ "${OUTPUT_DIR}" \ "${STACK_NAME}" \ "${TARGET_REGION}") OUTPUT_DIR="${NEW_OUTPUT_DIR}" ``` Update `README.md`'s `**Directory:**` field with the full absolute path of the renamed directory. If CFN deployment failed (Step 6 exceeded max retries), skip this step. Print a brief summary to the terminal: - Experiment output directory (with template ID) - CFN stack name and deployment status - Experiment template ID - Next step instruction ## Important Guidelines - **Scenario Library templates come from documentation.** Call `aws___read_documentation` on the scenario's doc URL (Step 1 table) before generating any files. The documentation is the only authoritative source. - **Never start the FIS experiment in this skill.** Starting the experiment is handled by `aws-fis-experiment-execute` or manually by the user. - **Validate resource-action compatibility BEFORE generating files** (Step 3). The most common source of wasted effort is deploying a template that targets an incompatible resource. - **Always deploy and validate.** Do not just generate files — deploy the CFN template and iterate until it succeeds (Step 6). The user should receive a working, deployed experiment template ready to start. - **Self-heal on CFN errors.** Read stack events, diagnose, fix the template, delete the failed stack, retry. Do not ask the user to fix CFN errors. - **Verify FIS action availability** (`aws fis list-actions` / `aws fis get-action`) before generating templates. Don't fabricate action IDs. - **Prefer `resourceArns` over `resourceTags` for targets.** Exceptions: `aws:elasticache:replicationgroup`, `aws:ec2:autoscaling-group`. Never combine `resourceArns` with `filters`. - **IAM policy must be least-privilege.** Only include permissions for the specific actions in the experiment. - **CFN template must be self-contained.** Deploy the CFN template and get a working experiment without any other steps. - **Sequential MCP calls.** All `aws___read_documentation` and `aws___search_documentation` calls must be sequential, never parallel. Retry up to 10 times on rate limit errors. - **Keep local files in sync.** After successful deployment, update README.md with real ARNs and stack outputs.


Security scan verdicts — ClawScan: suspicious | static analysis: suspicious | VirusTotal: clean | SkillSpector: suspicious (CRITICAL)