Skill: 1kalin/afrexai-devops-engine (version 1.0.0)

--- name: afrexai-devops-engine description: Complete DevOps & Platform Engineering system. CI/CD pipelines, infrastructure as code, container orchestration, observability, incident response, and SRE practices — all platforms, all clouds. metadata: {"clawdbot":{"emoji":"🔧","os":["linux","darwin","win32"]}} --- # DevOps & Platform Engineering Engine Complete system for building, deploying, operating, and observing production software. Covers the entire DevOps lifecycle — not just CI/CD, not just one cloud. ## Phase 1: Repository & Branch Strategy ### Git Flow Decision Matrix | Team Size | Release Cadence | Strategy | Branches | |-----------|----------------|----------|----------| | 1-3 | Continuous | Trunk-based | main + short-lived feature/ | | 4-15 | Weekly/biweekly | GitHub Flow | main + feature/ + PR | | 15+ | Scheduled releases | Git Flow | main + develop + feature/ + release/ + hotfix/ | | Regulated | Audited releases | Git Flow + tags | Above + signed tags + audit trail | ### Branch Protection Rules (Apply These) ```yaml # branch-protection.yml — document your rules main: required_reviews: 2 dismiss_stale_reviews: true require_codeowners: true require_status_checks: - ci/test - ci/lint - ci/security require_linear_history: true # No merge commits restrict_pushes: true # Only via PR require_signed_commits: false # Enable for regulated develop: required_reviews: 1 require_status_checks: - ci/test ``` ### Commit Convention Format: `<type>(<scope>): <description>` Types: `feat`, `fix`, `docs`, `style`, `refactor`, `perf`, `test`, `build`, `ci`, `chore` Breaking changes: `feat!: remove legacy API` or footer `BREAKING CHANGE: description` Enforce with commitlint + husky (Node) or pre-commit hooks. ## Phase 2: CI/CD Pipeline Architecture ### Pipeline Design Principles 1. **Build once, deploy everywhere** — same artifact through dev→staging→prod 2. **Fail fast** — cheapest checks first (lint→unit→integration→e2e) 3. **Hermetic builds** — no external state, reproducible from commit SHA 4. **Immutable artifacts** — never modify after build; tag with git SHA 5. **Parallelise independent stages** — test/lint/security scan simultaneously ### Universal Pipeline Template ```yaml # pipeline-stages.yml — adapt to your CI system stages: # Stage 1: Quality Gate (parallel, <2 min) lint: run: lint parallel: true timeout: 2m typecheck: run: tsc --noEmit parallel: true timeout: 2m security_scan: run: trivy, snyk, or semgrep parallel: true timeout: 3m # Stage 2: Test (parallel by type, <10 min) unit_tests: run: test --unit parallel: true coverage_threshold: 80% timeout: 5m integration_tests: run: test --integration parallel: true needs: [database_service] timeout: 10m # Stage 3: Build (<5 min) build: needs: [lint, typecheck, unit_tests] outputs: [docker_image, release_artifact] tag: "${GIT_SHA}" cache: [node_modules, .next/cache, target/] # Stage 4: Deploy Staging (auto) deploy_staging: needs: [build] environment: staging strategy: rolling smoke_test: true auto: true # Stage 5: E2E on Staging (<15 min) e2e_tests: needs: [deploy_staging] timeout: 15m retry: 1 artifacts: [screenshots, videos] # Stage 6: Deploy Production (manual gate or auto) deploy_prod: needs: [e2e_tests] environment: production strategy: canary # or blue-green approval: required # manual gate rollback_on_failure: true monitoring_window: 15m ``` ### CI Platform Cheat Sheet | Feature | GitHub Actions | GitLab CI | CircleCI | Jenkins | |---------|---------------|-----------|----------|---------| | Config file | `.github/workflows/*.yml` | `.gitlab-ci.yml` | `.circleci/config.yml` | `Jenkinsfile` | | Parallelism | `jobs.<id>` (automatic) | `stages` + `parallel` | `workflows` | `parallel` step | | Caching | `actions/cache` | `cache:` key | `save_cache/restore_cache` | Stash/unstash | | Secrets | Settings → Secrets | Settings → CI/CD → Variables | Project Settings → Env | Credentials plugin | | Matrix builds | `strategy.matrix` | `parallel:matrix` | `matrix` in workflows | `matrix` in pipeline | | Self-hosted | `runs-on: self-hosted` | GitLab Runner | `resource_class` | Default | | OIDC/Keyless | `permissions: id-token: write` | `id_tokens:` | OIDC context | Plugin | ### Caching Strategy ```yaml # Cache key patterns (ordered by specificity) cache_keys: # Exact match first - "deps-{{ runner.os }}-{{ hashFiles('**/lockfile') }}" # Partial match fallback - "deps-{{ runner.os }}-" # What to cache by stack node: [node_modules, .next/cache, .turbo] python: [.venv, .mypy_cache, .pytest_cache] rust: [target/, ~/.cargo/registry] go: [~/go/pkg/mod, ~/.cache/go-build] docker: [/tmp/.buildx-cache] # BuildKit layer cache ``` ### GitHub Actions Specific Patterns ```yaml # Reusable workflow (DRY across repos) # .github/workflows/reusable-deploy.yml on: workflow_call: inputs: environment: required: true type: string secrets: DEPLOY_KEY: required: true # Caller workflow jobs: deploy: uses: ./.github/workflows/reusable-deploy.yml with: environment: production secrets: inherit ``` ```yaml # Path-based triggers (monorepo) on: push: paths: - 'packages/api/**' - 'shared/**' # Skip CI for docs-only changes pull_request: paths-ignore: - '**.md' - 'docs/**' ``` ```yaml # Concurrency (cancel in-progress on new push) concurrency: group: ${{ github.workflow }}-${{ github.ref }} cancel-in-progress: true ``` ## Phase 3: Container Strategy ### Dockerfile Best Practices ```dockerfile # Multi-stage build template # Stage 1: Build FROM node:20-alpine AS builder WORKDIR /app COPY package.json package-lock.json ./ RUN npm ci --production=false # Install all deps for build COPY . . RUN npm run build # Stage 2: Production FROM node:20-alpine AS production RUN addgroup -g 1001 app && adduser -u 1001 -G app -s /bin/sh -D app WORKDIR /app COPY --from=builder --chown=app:app /app/dist ./dist COPY --from=builder --chown=app:app /app/node_modules ./node_modules COPY --from=builder --chown=app:app /app/package.json ./ USER app EXPOSE 3000 HEALTHCHECK --interval=30s --timeout=3s --retries=3 \ CMD wget -qO- http://localhost:3000/health || exit 1 CMD ["node", "dist/index.js"] ``` ### Image Size Reduction Checklist - [ ] Use alpine or distroless base images - [ ] Multi-stage builds (build deps not in final image) - [ ] `.dockerignore` excludes: `.git`, `node_modules`, `*.md`, tests, docs - [ ] Combine RUN commands (fewer layers) - [ ] Clean package manager cache in same RUN (`rm -rf /var/cache/apk/*`) - [ ] No dev dependencies in production stage - [ ] Pin base image SHA: `FROM node:20-alpine@sha256:abc123...` ### Container Security Scan ```bash # Trivy (recommended — free, fast) trivy image myapp:latest --severity HIGH,CRITICAL trivy fs . --security-checks vuln,secret,config # Scan in CI before push # Fail pipeline if CRITICAL vulnerabilities found trivy image --exit-code 1 --severity CRITICAL myapp:${GIT_SHA} ``` ### Docker Compose for Local Dev ```yaml # docker-compose.yml — local development stack services: app: build: context: . target: builder # Use build stage for hot reload volumes: - .:/app - /app/node_modules # Don't override node_modules ports: - "3000:3000" environment: - DATABASE_URL=postgres://user:pass@db:5432/app - REDIS_URL=redis://cache:6379 depends_on: db: condition: service_healthy db: image: postgres:16-alpine volumes: - pgdata:/var/lib/postgresql/data environment: POSTGRES_USER: user POSTGRES_PASSWORD: pass POSTGRES_DB: app healthcheck: test: ["CMD-SHELL", "pg_isready -U user"] interval: 5s timeout: 3s retries: 5 cache: image: redis:7-alpine ports: - "6379:6379" volumes: pgdata: ``` ## Phase 4: Infrastructure as Code ### IaC Decision Matrix | Tool | Best For | State | Language | Learning Curve | |------|----------|-------|----------|----------------| | Terraform/OpenTofu | Multi-cloud, cloud-agnostic | Remote (S3, GCS) | HCL | Medium | | Pulumi | Devs who prefer real code | Remote | TS/Python/Go | Low (if you code) | | AWS CDK | AWS-only shops | CloudFormation | TS/Python | Medium | | Ansible | Config management, server setup | Stateless | YAML | Low | | Helm | Kubernetes deployments | Tiller/OCI | YAML+Go templates | Medium | ### Terraform Project Structure ``` infrastructure/ ├── modules/ # Reusable components │ ├── vpc/ │ │ ├── main.tf │ │ ├── variables.tf │ │ └── outputs.tf │ ├── ecs-service/ │ └── rds/ ├── environments/ │ ├── dev/ │ │ ├── main.tf # Calls modules with dev params │ │ ├── terraform.tfvars │ │ └── backend.tf # Dev state bucket │ ├── staging/ │ └── prod/ ├── .terraform-version # Pin terraform version └── .tflint.hcl ``` ### Terraform Safety Rules 1. **Always `plan` before `apply`** — review every change 2. **Remote state with locking** — S3 + DynamoDB or GCS + locking 3. **State never in git** — contains secrets (DB passwords, keys) 4. **Import existing resources** before managing them — don't recreate 5. **Use `prevent_destroy`** on critical resources (databases, S3 buckets) 6. **Tag everything** — `environment`, `team`, `cost-center`, `managed-by: terraform` 7. **`terraform fmt`** in CI — consistent formatting ```hcl # backend.tf — remote state with locking terraform { backend "s3" { bucket = "mycompany-terraform-state" key = "prod/main.tfstate" region = "eu-west-1" encrypt = true dynamodb_table = "terraform-locks" } } # Protect critical resources resource "aws_rds_instance" "main" { # ... lifecycle { prevent_destroy = true } } ``` ### Environment Promotion Pattern ``` ┌──────────────────┐ terraform plan ──►│ Review in PR │ └────────┬─────────┘ │ merge ┌────────▼─────────┐ auto-apply ──────►│ Dev │──► smoke tests └────────┬─────────┘ │ promote ┌────────▼─────────┐ manual approve ──►│ Staging │──► integration tests └────────┬─────────┘ │ promote (manual gate) ┌────────▼─────────┐ manual approve ──►│ Production │──► monitoring window └──────────────────┘ ``` ## Phase 5: Kubernetes Operations ### K8s Resource Templates ```yaml # deployment.yml — production-ready template apiVersion: apps/v1 kind: Deployment metadata: name: myapp labels: app: myapp version: "1.0.0" spec: replicas: 3 strategy: type: RollingUpdate rollingUpdate: maxSurge: 1 maxUnavailable: 0 # Zero-downtime selector: matchLabels: app: myapp template: metadata: labels: app: myapp spec: securityContext: runAsNonRoot: true runAsUser: 1000 containers: - name: myapp image: myregistry/myapp:abc123 # Git SHA tag ports: - containerPort: 3000 resources: requests: cpu: 100m memory: 128Mi limits: cpu: 500m memory: 512Mi livenessProbe: httpGet: path: /health port: 3000 initialDelaySeconds: 10 periodSeconds: 10 failureThreshold: 3 readinessProbe: httpGet: path: /ready port: 3000 initialDelaySeconds: 5 periodSeconds: 5 env: - name: DATABASE_URL valueFrom: secretKeyRef: name: myapp-secrets key: database-url topologySpreadConstraints: - maxSkew: 1 topologyKey: topology.kubernetes.io/zone whenUnsatisfiable: DoNotSchedule ``` ```yaml # hpa.yml — autoscaling apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler metadata: name: myapp spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: myapp minReplicas: 3 maxReplicas: 20 metrics: - type: Resource resource: name: cpu target: type: Utilization averageUtilization: 70 - type: Resource resource: name: memory target: type: Utilization averageUtilization: 80 behavior: scaleDown: stabilizationWindowSeconds: 300 # 5 min cooldown policies: - type: Pods value: 1 periodSeconds: 60 # Scale down 1 pod per minute max ``` ### Helm Chart Checklist - [ ] `values.yaml` with sensible defaults (works out of the box) - [ ] Resource requests AND limits set - [ ] Health/readiness probes defined - [ ] PodDisruptionBudget (minAvailable: 1 or maxUnavailable: 25%) - [ ] NetworkPolicy (deny all, allow specific) - [ ] ServiceAccount (not default) - [ ] Secrets via external-secrets-operator or sealed-secrets (not plain) - [ ] `helm lint` and `helm template` in CI - [ ] NOTES.txt with post-install instructions ### kubectl Cheat Sheet ```bash # Debugging kubectl get pods -l app=myapp -o wide # Pod status + node kubectl describe pod <pod> # Events, conditions kubectl logs <pod> --tail=100 -f # Stream logs kubectl logs <pod> --previous # Crashed container logs kubectl exec -it <pod> -- /bin/sh # Shell into pod kubectl top pods -l app=myapp # Resource usage # Rollouts kubectl rollout status deployment/myapp # Watch rollout kubectl rollout history deployment/myapp # Revision history kubectl rollout undo deployment/myapp # Rollback to previous kubectl rollout undo deployment/myapp --to-revision=3 # Specific # Scaling kubectl scale deployment/myapp --replicas=5 # Manual scale kubectl autoscale deployment/myapp --min=3 --max=10 --cpu-percent=70 # Context management kubectl config get-contexts # List clusters kubectl config use-context prod-cluster # Switch kubectl config set-context --current --namespace=myapp # Set namespace ``` ## Phase 6: Deployment Strategies ### Strategy Decision Matrix | Strategy | Risk | Speed | Rollback | Cost | Best For | |----------|------|-------|----------|------|----------| | Rolling | Low-Med | Fast | Slow (re-roll) | None | Standard deployments | | Blue-Green | Low | Instant | Instant (switch) | 2x infra | Critical services, zero-downtime | | Canary | Very Low | Slow | Instant (route 0%) | Minimal | High-traffic, risky changes | | Feature Flag | Very Low | Instant | Instant (toggle) | None | Gradual rollout, A/B testing | | Recreate | High | Fast | Slow | None | Dev/staging, stateful apps | ### Canary Deployment Workflow ``` 1. Deploy canary (1 pod with new version) 2. Route 5% traffic → canary 3. Monitor for 5 minutes: - Error rate < baseline + 0.1%? - p99 latency < baseline + 50ms? - No new error types? 4. If healthy → 25% → monitor 10 min 5. If healthy → 50% → monitor 10 min 6. If healthy → 100% (full rollout) 7. If ANY check fails → route 0% to canary → rollback → alert Automation: Argo Rollouts, Flagger, or Istio + custom controller ``` ### Rollback Checklist When a deployment goes wrong: 1. **Immediate**: Route traffic away from new version (canary→0%, blue-green→switch) 2. **If rolling**: `kubectl rollout undo` or redeploy previous SHA 3. **Check**: Are database migrations backward-compatible? (If not, you have a bigger problem) 4. **Verify**: Rollback successful? Check error rates, latency 5. **Communicate**: Post in #incidents, update status page 6. **Investigate**: Don't re-deploy until root cause found ### Database Migration Safety ``` RULE: Migrations must be backward-compatible with the PREVIOUS version. (Because during rolling deploy, both versions run simultaneously) Safe migration pattern: v1: Add new column (nullable, with default) v2: Backfill data, start writing to new column v3: Make new column required, stop writing old column v4: Drop old column (after v3 is fully deployed) NEVER in one deploy: ❌ Rename column ❌ Change column type ❌ Drop column still read by current version ❌ Add NOT NULL without default ``` ## Phase 7: Observability Stack ### Three Pillars + Bonus | Pillar | What | Tools | Priority | |--------|------|-------|----------| | **Metrics** | Numeric measurements over time | Prometheus, Datadog, CloudWatch | 1 (start here) | | **Logs** | Event records | ELK, Loki, CloudWatch Logs | 2 | | **Traces** | Request flow across services | Jaeger, Tempo, X-Ray, Honeycomb | 3 | | **Profiling** | CPU/memory hot paths | Pyroscope, Parca | 4 (when optimizing) | ### Key Metrics to Track ```yaml # RED Method (request-driven services) rate: # Requests per second errors: # Failed requests per second duration: # Latency distribution (p50, p95, p99) # USE Method (infrastructure/resources) utilization: # % of resource in use (CPU, memory, disk) saturation: # Queue depth, pending work errors: # Resource errors (OOM, disk full) # Business Metrics (most important!) signups_per_hour: checkout_completion_rate: api_calls_by_customer: revenue_per_minute: ``` ### Alerting Rules ```yaml # alerting-rules.yml alerts: # Symptom-based (good — tells you users are impacted) - name: HighErrorRate condition: "error_rate_5xx > 1% for 5m" severity: critical runbook: docs/runbooks/high-error-rate.md notify: [pagerduty, slack-incidents] - name: HighLatency condition: "p99_latency > 2s for 5m" severity: warning runbook: docs/runbooks/high-latency.md notify: [slack-incidents] # Cause-based (supplementary — helps diagnose) - name: PodCrashLooping condition: "pod_restart_count increase > 3 in 10m" severity: warning notify: [slack-platform] - name: DiskSpaceWarning condition: "disk_usage > 80%" severity: warning notify: [slack-platform] - name: CertificateExpiring condition: "cert_expiry_days < 14" severity: warning notify: [slack-platform] # Alert rules: # 1. Every alert must have a runbook link # 2. Every alert must be actionable (if you can't do anything, remove it) # 3. Critical = wake someone up. Warning = check next business day. # 4. Review alerts monthly — archive unused, tune noisy ones ``` ### Structured Logging Standard ```json { "timestamp": "2026-02-16T05:00:00.000Z", "level": "error", "service": "api", "trace_id": "abc123", "span_id": "def456", "method": "POST", "path": "/api/orders", "status": 500, "duration_ms": 342, "user_id": "usr_789", "error": { "type": "DatabaseError", "message": "connection timeout", "stack": "..." }, "context": { "order_id": "ord_123", "payment_method": "card" } } ``` **Log level guide:** - `error`: Something failed, needs attention - `warn`: Unexpected but handled (retry succeeded, fallback used) - `info`: Business events (order placed, user signed up, deploy started) - `debug`: Technical detail (query executed, cache hit/miss) — OFF in prod ### Dashboard Template Every service dashboard should have: ``` Row 1: Traffic Overview - Request rate (per endpoint) - Error rate (4xx, 5xx separate) - Active users / connections Row 2: Performance - p50, p95, p99 latency - Throughput - Apdex score Row 3: Resources - CPU utilization (per pod/instance) - Memory usage (vs limit) - Disk I/O / Network I/O Row 4: Business - Revenue per minute (if applicable) - Conversion funnel - Queue depth / processing lag Row 5: Dependencies - Database query latency + connection pool - External API latency + error rate - Cache hit rate ``` ## Phase 8: Incident Response ### Severity Levels | Level | Definition | Response Time | Example | |-------|-----------|---------------|---------| | SEV-1 | Complete outage, revenue impact | 15 min | Site down, payments failing | | SEV-2 | Major feature broken, workaround exists | 30 min | Search broken, checkout slow | | SEV-3 | Minor feature broken, low impact | 4 hours | Admin panel bug, non-critical API | | SEV-4 | Cosmetic / no user impact | Next sprint | Typo, minor UI glitch | ### Incident Workflow ``` 1. DETECT (automated or reported) → Alert fires / user reports issue → Create incident channel: #inc-YYYY-MM-DD-description 2. TRIAGE (first 5 minutes) → Assign Incident Commander (IC) → Determine severity level → Post initial assessment in channel → Update status page (if customer-facing) 3. MITIGATE (focus on stopping the bleeding) → Can we rollback? → Do it → Can we scale up? → Do it → Can we feature-flag disable? → Do it → DON'T debug root cause yet — restore service first 4. RESOLVE → Confirm service restored (metrics, customer reports) → Communicate resolution to stakeholders → Update status page 5. POST-MORTEM (within 48 hours) → Blameless — focus on systems, not people → Timeline of events → Root cause analysis (5 Whys) → Action items with owners and deadlines → Share with team ``` ### Post-Mortem Template ```markdown # Incident Post-Mortem: [Title] **Date:** YYYY-MM-DD **Duration:** Xh Ym **Severity:** SEV-X **Incident Commander:** [name] **Author:** [name] ## Summary [1-2 sentence summary of what happened and impact] ## Impact - Users affected: [number/percentage] - Revenue impact: [if applicable] - Duration: [start to full resolution] ## Timeline (all times UTC) | Time | Event | |------|-------| | 14:00 | Deploy v2.3.1 begins | | 14:05 | Error rate spikes to 15% | | 14:07 | Alert fires, IC paged | | 14:12 | Rollback initiated | | 14:15 | Service restored | ## Root Cause [Technical explanation — what actually broke and why] ## Contributing Factors - [Factor 1 — e.g., migration not tested with production data volume] - [Factor 2 — e.g., canary deployment not configured for this service] ## What Went Well - [Fast detection — alert fired within 2 minutes] - [Clear runbook — IC knew rollback procedure] ## What Went Wrong - [No canary — went straight to 100% rollout] - [Migration was not backward-compatible] ## Action Items | Action | Owner | Due | Priority | |--------|-------|-----|----------| | Add canary to deployment | @engineer | YYYY-MM-DD | P1 | | Add migration backward-compat check | @engineer | YYYY-MM-DD | P1 | | Update runbook for this service | @sre | YYYY-MM-DD | P2 | ## Lessons Learned [Key takeaways for the team] ``` ### On-Call Best Practices ```yaml on_call: rotation: weekly handoff: Monday 10:00 (overlap 1h with previous) escalation: - primary: respond within 15 min - secondary: auto-page if no ack in 15 min - manager: auto-page if no ack in 30 min expectations: - Laptop + internet within reach - Respond to page within 15 minutes - Follow runbook first, improvise second - Escalate early — "I don't know" is fine - Update incident channel every 15 min during active incident wellness: - No more than 1 week in 4 on-call - Comp time after major incidents - Toil budget: <30% of on-call time should be toil - Quarterly review: are we paging too much? ``` ## Phase 9: Security Hardening ### Security Checklist (CI Pipeline) ```yaml security_gates: # Pre-commit - tool: gitleaks / trufflehog what: Secret detection in code block: true # Build - tool: semgrep / CodeQL what: Static analysis (SAST) block: critical findings - tool: npm audit / pip audit / cargo audit what: Dependency vulnerabilities (SCA) block: critical/high # Container - tool: trivy / grype what: Image vulnerability scan block: critical - tool: hadolint what: Dockerfile best practices block: error level # Deploy - tool: checkov / tfsec what: IaC security scan block: high findings # Runtime - tool: falco / sysdig what: Runtime anomaly detection alert: true ``` ### Secrets Management Decision | Method | Security | Complexity | Best For | |--------|----------|------------|----------| | CI/CD env vars | Basic | Low | Small teams, non-critical | | AWS Secrets Manager / GCP Secret Manager | High | Medium | Cloud-native apps | | HashiCorp Vault | Very High | High | Multi-cloud, strict compliance | | SOPS + git | Good | Low | GitOps workflows | | External Secrets Operator | High | Medium | Kubernetes + cloud secrets | **Rules:** - Rotate secrets every 90 days minimum - Different secrets per environment (dev ≠ staging ≠ prod) - Audit all secret access - Never log secrets — mask in CI output - Use OIDC/keyless auth where possible (no long-lived tokens) ### Network Security Baseline ``` 1. Default deny all — explicitly allow what's needed 2. TLS everywhere — including internal service-to-service 3. No public IPs on internal services — use load balancers / API gateways 4. WAF on public endpoints — OWASP Top 10 rules minimum 5. Rate limiting on all APIs — prevent abuse and DDoS 6. DNS for service discovery — never hardcode IPs 7. VPN or zero-trust for admin access — no SSH from internet 8. Network policies in K8s — pods can't talk to everything 9. Egress control — services should only reach what they need 10. Certificate auto-renewal — cert-manager or ACM ``` ## Phase 10: SRE Practices ### SLO Framework ```yaml # Define SLOs for every user-facing service service: checkout-api slos: availability: target: 99.95% # 4.38 hours downtime/year window: 30d rolling measurement: "successful_requests / total_requests" latency: target: 99% # 99% of requests under threshold threshold: 500ms # p99 < 500ms window: 30d rolling freshness: target: 99.9% # Data updated within SLA threshold: 5m window: 30d rolling error_budget: monthly_budget: 0.05% # ~21.6 minutes burn_rate_alert: fast: 14.4x # Budget consumed in 1 hour → page slow: 3x # Budget consumed in 10 hours → ticket policy: budget_exhausted: - freeze non-critical deploys - redirect eng effort to reliability - review in weekly SRE sync ``` ### Toil Reduction ``` Toil = manual, repetitive, automatable, reactive, no lasting value Track toil: - Log manual interventions for 2 weeks - Categorize: deployment, scaling, cert renewal, data fixes, permissions - Prioritize: frequency × time × frustration Target: <30% of engineering time on toil If toil > 50%: stop feature work, automate the top 3 toil items Common toil automation: Manual deploys → CI/CD pipeline Certificate renewal → cert-manager / ACM Scaling up/down → HPA / auto-scaling groups Permission requests → Self-service IAM with approval Data fixes → Admin API / scripts Dependency updates → Renovate / Dependabot Flaky test management → Auto-quarantine + ticket ``` ### Capacity Planning ```yaml capacity_review: frequency: monthly inputs: - current_utilization: "CPU, memory, disk, network per service" - growth_rate: "request rate trend over 90 days" - planned_events: "launches, marketing campaigns, seasonal peaks" - headroom_target: 30% # Don't run above 70% sustained formula: needed_capacity: "current_usage × (1 + growth_rate) × (1 + headroom)" lead_time: "14 days for cloud, 60+ days for hardware" actions: - "If utilization > 70%: plan scaling within 2 weeks" - "If utilization > 85%: emergency scaling NOW" - "If utilization < 30%: rightsize down (save money)" ``` ## Phase 11: Cost Optimization ### Cloud Cost Rules ``` 1. Right-size first — most instances are overprovisioned Check: actual CPU/memory usage vs provisioned (CloudWatch, Datadog) Action: downsize to next tier that maintains 70% headroom 2. Reserved capacity for baseline — spot/preemptible for burst Pattern: 60% reserved + 30% on-demand + 10% spot Savings: 40-70% on reserved vs on-demand 3. Auto-scale to zero when possible - Dev/staging environments: scale down nights + weekends - Serverless for bursty workloads (Lambda, Cloud Functions) 4. Delete zombie resources monthly - Unattached EBS volumes - Old snapshots (>90 days, not tagged for retention) - Unused load balancers - Orphaned Elastic IPs 5. Storage tiering - Hot: SSD (frequently accessed) - Warm: HDD (monthly access) - Cold: S3 Glacier / Archive (yearly access) - Auto-lifecycle policies on S3 buckets 6. Tag everything — untagged = untracked = wasted Required tags: environment, team, service, cost-center Weekly report: cost by tag, highlight untagged resources ``` ### Monthly Cost Review Template ```markdown ## Cloud Cost Review — [Month YYYY] ### Summary - Total spend: $X,XXX (vs budget: $X,XXX) - MoM change: +X% ($XXX) - Top 3 cost drivers: [service1, service2, service3] ### By Service | Service | Cost | % of Total | MoM Change | Action | |---------|------|-----------|------------|--------| | EKS | $XXX | XX% | +X% | Right-size node group | | RDS | $XXX | XX% | 0% | Consider reserved | | S3 | $XXX | XX% | +X% | Add lifecycle rules | ### Optimization Actions Taken - [Action 1]: Saved $XXX/mo - [Action 2]: Saved $XXX/mo ### Next Month Actions - [ ] [Action with estimated savings] ``` ## DevOps Maturity Assessment Score your team (1-5 per dimension): | Dimension | 1 (Ad-hoc) | 3 (Defined) | 5 (Optimized) | |-----------|-----------|-------------|----------------| | **CI/CD** | Manual deploy | Automated pipeline, manual gate | Full auto with canary, <15 min to prod | | **IaC** | Click-ops console | Some Terraform, manual tweaks | 100% IaC, GitOps, drift detection | | **Monitoring** | Check when broken | Dashboards + basic alerts | SLOs, error budgets, auto-remediation | | **Incident** | Panic + SSH | Runbooks, on-call rotation | Blameless postmortems, chaos engineering | | **Security** | Annual audit | CI scanning, secret manager | Shift-left, runtime detection, zero-trust | | **Cost** | Surprise bills | Monthly review, some reservations | Real-time tracking, auto-optimization | **Score interpretation:** - 6-12: Foundations needed — focus on CI/CD and basic monitoring - 13-20: Growing — add IaC and incident process - 21-26: Mature — optimize with SRE practices and cost management - 27-30: Elite — focus on chaos engineering and developer experience ## Natural Language Commands Say things like: - "Set up CI/CD for my Node.js project" - "Create a Dockerfile for my Python API" - "Write Terraform for an ECS service with RDS" - "Design a monitoring dashboard for my service" - "Help me write a post-mortem for yesterday's outage" - "Review my Kubernetes deployment for production readiness" - "What deployment strategy should I use?" - "Help me set up alerting rules" - "Create an incident response runbook for database failures" - "Audit my cloud costs and suggest optimizations" - "Assess our DevOps maturity" - "Set up secret management for our CI pipeline"


Security scan verdicts — ClawScan: clean | static analysis:  | VirusTotal: clean | SkillSpector: clean (LOW)