Skill: charlie-morrison/k8s-cost-optimizer (version 1.0.0)

--- name: k8s-cost-optimizer description: Find and rank Kubernetes cost-saving opportunities from kubectl, metrics-server, kube-state-metrics, and cloud billing. Identifies overprovisioned CPU/memory requests and limits, idle namespaces and workloads, oversized PersistentVolumes, unused LoadBalancer services, expensive node types, missing HorizontalPodAutoscalers, and clusters that haven't adopted spot/preemptible/Graviton nodes. Outputs a ranked list of recommendations with $/month savings estimates and ready-to-apply YAML patches. Covers EKS, GKE, and AKS specifics including instance pricing, savings plans, committed-use discounts, and reservation strategies. Use when asked to cut a Kubernetes cloud bill, right-size workloads, plan a spot migration, build a FinOps report, or tune HPA settings. Triggers on "kubernetes cost", "k8s cost", "eks cost", "gke cost", "aks cost", "right-size", "rightsize", "kubecost", "opencost", "vpa", "hpa", "spot instances", "preemptible", "savings plan", "node pool", "pod requests", "finops". metadata: tags: ["kubernetes", "k8s", "eks", "gke", "aks", "finops", "cost-optimization", "rightsizing", "autoscaling", "spot-instances", "kubecost", "opencost", "platform-engineering", "devops"] --- # Kubernetes Cost Optimizer Audit a Kubernetes cluster (or fleet) and produce a ranked list of cost-saving actions with concrete dollar estimates. Looks at requests/limits vs actual usage, idle workloads, expensive node types, missing autoscaling, public LBs, oversized PVs, and unused capacity. Acts as a senior FinOps engineer who has cut six- and seven-figure cloud bills without breaking workloads. ## Usage Invoke this skill when a Kubernetes bill is too high, when a quarterly FinOps review is due, or when leadership has asked for "30% off the cloud." **Basic invocation:** > Audit my EKS cluster for cost savings > Cut my GKE bill — here's kubectl top + node list > What's the highest-ROI optimization I can ship this week? **With context:** > Here's metrics-server data for 30 days, the node list, and the AWS bill > I have 14 namespaces — which ones are idle? > We're 100% on-demand m5 nodes — what's the spot migration plan? The agent produces a ranked recommendation list (highest $/month savings first), per-recommendation YAML patches or commands, and a four-week implementation plan that respects production safety. ## How It Works ### Step 1: Data Collection Cost optimization without data is guesswork. The agent collects from four sources and joins them: | Source | What It Provides | How To Pull | |--------|------------------|-------------| | **kubectl + metrics-server** | Real CPU/memory usage per pod, per node | `kubectl top pods -A`, `kubectl top nodes` | | **kube-state-metrics / Prometheus** | Requests, limits, replicas, deployment-level history | PromQL: [REDACTED_SECRET], 30-day window | | **Cloud billing** | $/node-hour, instance type, region, sustained-use | AWS Cost Explorer, GCP billing export, Azure Cost Management | | **Cluster object inventory** | Namespaces, services, PVCs, ingress, jobs, cronjobs | `kubectl get all,pvc,svc -A -o json` | Data **window** matters. The agent prefers 30 days; 7 days for fast-moving clusters; 90 days for capacity planning. Anything under 7 days is too short — diurnal and weekly patterns dominate the noise. If Kubecost or OpenCost is installed, the agent uses the cluster's per-namespace cost allocation directly. Otherwise it computes allocations from node price × pod-share-of-node. ### Step 2: The Cost Recommendation Catalog The agent runs the cluster against a fixed set of recommendation **types**, each with a detection rule and a savings formula. **C1. Overprovisioned CPU requests** ``` Detection: for each container, p99(cpu_usage over 30d) < 0.50 * cpu_request AND container has >7 days of data AND deployment is not a known-bursty type (cron, batch, init) Savings estimate: ($/cpu-hour for the node pool) × (request - p99usage) × 24 × 30 × replicas Action: patch container.resources.requests.cpu down to ceil(p95 × 1.3) ``` **C2. Overprovisioned memory requests** ``` Detection: p99(memory_working_set over 30d) < 0.50 * memory_request Savings: ($/GiB-hour for the node pool) × (request - p99usage) × 24 × 30 × replicas Action: patch container.resources.requests.memory down to ceil(p99 × 1.25) NOTE: never set requests below working-set-p99 — OOMKills kill the savings ``` **C3. Limits == requests (no burst)** ``` Detection: cpu_limit == cpu_request for stateless workloads (typical anti-pattern: "treat limits as guaranteed quota") Savings: None directly — but C1 dominates after limits are unblocked Action: raise limits or remove (for cpu); keep limits for memory ``` **C4. Idle namespace** ``` Detection: sum(p95 cpu over 30d) across all pods in ns < 0.05 cores AND sum(p95 memory) < 200 MiB AND no recent kubectl apply (last_modified > 30 days) Savings: All allocated capacity (request × node $) Action: warn → tag → archive (Helm release deleted, namespace archived) ``` **C5. Idle deployment / statefulset** ``` Detection: replicas > 0 AND p99(cpu) < 0.02 cores AND request_count == 0 over 30d (request_count from ingress-controller or service mesh) Savings: replicas × pod_cost / month Action: scale to zero (KEDA cron, or just `kubectl scale --replicas=0`) ``` **C6. Oversized PersistentVolume** ``` Detection: for each PVC, kubelet_volume_stats_used / capacity < 0.3 AND age > 30 days Savings: ($/GB-month for storage class) × (capacity - used × 1.5) Action: - On EKS gp3: shrink not supported. Migrate via snapshot → smaller PV. - On GKE pd-balanced: same — snapshot migration. - On AKS managed-disks: same. Plan downtime. ``` **C7. Unused LoadBalancer service** ``` Detection: Service type=LoadBalancer AND no NetworkPolicy hits AND no ingress traffic in 30d (cloud LB metrics) Savings: AWS NLB: ~$22/mo + $0.006/LCU-hr → $25-50/mo typical GCP LB: ~$18/mo per forwarding rule Azure LB: ~$25/mo standard tier Action: delete service or convert to ClusterIP behind a shared ingress ``` **C8. Expensive node type** ``` Detection: Node pool uses x86 on a workload that's arch-independent AND no GPU/specialized requirement AND newer-gen / Graviton / Tau alternative is cheaper per CPU-hour Savings: AWS: m5 → m7g (Graviton) ~20% cheaper, similar perf GCP: n2 → t2d (Tau AMD) ~28% cheaper, comparable perf Azure: Dsv3 → Dpdsv5 (Arm) ~20% cheaper Action: add Arm/AMD node pool, taint, set tolerations on workloads, recompile multi-arch images (most public images are already multi-arch) ``` **C9. Missing HorizontalPodAutoscaler** ``` Detection: Deployment with stable replica count > 3 AND p95/p50 cpu ratio > 2.5x (variance) AND no HPA / KEDA / Karpenter scaler attached Savings: (max_replicas - avg_replicas) × pod_cost typical: 30-60% of deployment's compute Action: emit HPA YAML targeted at p50 of recent CPU ``` **C10. No spot / preemptible / Spot VM adoption** ``` Detection: Cluster is 100% on-demand AND has stateless workloads (Deployments without local volume requirements) AND tolerates interruption (replicas > 1, restart-safe) Savings: AWS Spot: 60-90% off on-demand GCP Preemptible: 60-91% off (24h max lifetime) GCP Spot VMs: 60-91% off (no time limit, lower preemption rate) Azure Spot: 60-90% off (eviction subject to capacity) Action: - Tag stateless workloads with affinity for spot pool - Add a managed-on-demand fallback pool sized to baseline - Use Karpenter (AWS) / cluster-autoscaler with multiple ASGs / Azure Spot Priority Mix ``` **C11. Missing pod disruption budgets / wrong topology spread** ``` Detection: Workload running on spot but no PDB OR PDB minAvailable >= replicas (always blocks eviction) Savings: Indirect — wrong PDB blocks spot benefits Action: set PDB minAvailable = replicas - 1 (or maxUnavailable = 25%) add topologySpreadConstraints across zones ``` **C12. Stale CronJobs and Jobs** ``` Detection: Job/CronJob age > 90 days, never succeeded recently OR successfulJobsHistoryLimit unset (default 3 retains 3, no cleanup) Savings: Small per-cluster but accumulates: PVC retention, image-pull, scheduling churn Action: set ttlSecondsAfterFinished, prune old job objects, remove dead cronjobs ``` **C13. Image pull cost (GCR / ECR / ACR cross-region)** ``` Detection: Cluster in region X pulls from registry in region Y → cross-region egress charges Savings: Egress cost (typically $0.02-0.09/GB) × image-size × pod-restarts Action: replicate registry into cluster region enable image-pull-policy: IfNotPresent pre-pull common images via DaemonSet ``` **C14. Reserved capacity / Savings Plans / CUDs not purchased** ``` Detection: Stable baseline > 70% of cluster running 24/7 for 90+ days AND no savings plans or CUDs in cloud account Savings: AWS Compute Savings Plan: 30-66% off depending on commitment GCP CUD: 20-57% off (1-year and 3-year) Azure Reservations: 30-72% off Action: size commitment to baseline (not peak) use Savings Plans (flexible) over RI (rigid) on AWS ``` **C15. Logs / metrics / traces ingestion cost** ``` Detection: Cluster sends 100% of logs to managed observability (Datadog, NewRelic, Splunk) AND log volume > 1 TB/month AND no sampling on chatty namespaces (kube-system, ingress-controller) Savings: Often the largest single line item — Datadog logs at $0.10/GB ingested adds up fast; one chatty pod can cost $5,000/mo Action: - Drop INFO logs from kube-system, controller-manager - Sample debug logs at 1% - Route to S3 + Athena for cold logs ``` ### Step 3: Ranking The Recommendations Recommendations are ordered by **expected savings × implementation safety / implementation effort**. The agent renders this as a table: ``` Rank Type Description $/mo saved Effort Risk 1 C1 Right-size payments-api requests $4,200 2 hrs Low 2 C10 Migrate stateless to spot (Karpenter) $3,800 2 days Med 3 C8 Move backend pool to Graviton $2,100 1 day Low 4 C15 Drop kube-system DEBUG logs in Datadog $1,900 1 hr Low 5 C7 Delete 4 unused NLBs $190 30 min Low 6 C9 HPA on the api-gateway deployment $850 2 hrs Low 7 C6 Shrink 3 oversized PVs $310 4 hrs Med ... etc ``` **Effort** is hours-of-engineering. **Risk** is Low/Med/High based on user-facing blast radius. ### Step 4: Right-Sizing Methodology C1 and C2 dominate most clusters' savings. The agent applies a careful methodology: ``` 1. WINDOW 30 days minimum. Cover at least one full release cycle. Exclude windows containing incidents (skewed up) or maintenance windows (skewed down). 2. PERCENTILE Memory: p99 working_set × 1.25 = new request CPU: p95 × 1.3 = new request Why different: memory OOMKills are catastrophic, CPU throttling is recoverable. 3. FLOOR Never below 50m CPU / 64Mi memory — startup probes need headroom. For Java/JVM: use the JVM's heap-max as memory floor. For Go: 2x the runtime's resident set. 4. LIMITS CPU limits: REMOVE for most workloads (CFS throttling causes more latency than it saves money). Memory limits: KEEP at request × 1.5 — OOM is a controlled failure. 5. STAGED ROLLOUT Apply to a canary deployment (1 of N replicas) for 48h. Monitor: error rate, latency p99, restart count, OOMKill count. If clean, roll out to remaining replicas. If regression, revert; widen the window or raise the percentile. 6. VPA RECOMMENDATION MODE For unfamiliar workloads, run VPA in `recommendOnly` mode for 7+ days. Use its target as a sanity-check on the agent's calculation. ``` ### Step 5: Cloud-Specific Specifics **EKS (AWS):** - **Karpenter** beats cluster-autoscaler on EKS for cost: provisions exactly the right node, mixes instance types, adopts spot natively. Migrating to Karpenter is often a 15-25% saving on its own. - **Graviton (m7g, c7g, r7g)**: ~20% cheaper than equivalent x86. Multi-arch images are required; check `docker manifest inspect`. - **Spot via Karpenter**: set `karpenter.sh/capacity-type: spot` in the NodePool; add an on-demand fallback pool. - **EKS control-plane** is $0.10/hr per cluster — cluster consolidation is sometimes the right move (one prod, one staging, one dev rather than per-team clusters). - **AWS Compute Savings Plan**: covers EC2, Fargate, Lambda; flexible across instance families. Buy at baseline, not peak. - **EBS gp3** is cheaper than gp2 at equal IOPS — easy migration via volume modify. **GKE:** - **Autopilot** mode: no node management, charged per pod-second. Often cheaper for low-utilization clusters; more expensive for dense clusters. - **Spot VMs** preferred over old Preemptible (no 24h cap, lower preemption rate). - **CUDs** apply to vCPU and memory committed for 1y or 3y; flexible across instance families with the new "flex CUD." - **T2D / Tau**: AMD-based, ~28% cheaper for general-purpose workloads. - **GKE control-plane** is free for one zonal cluster per project; regional clusters cost extra. - **Cluster Autoscaler** scales node pools but cannot create new pools — pre-create spot/Graviton pools with appropriate taints. **AKS:** - **AKS control-plane** is free (Azure pays); only nodes are billed. - **Spot node pools** require explicit provisioning; eviction policy `Deallocate` keeps disks but releases compute. - **Reserved Instances** apply at the VM-family level (Dsv5, Esv5); commit to baseline. - **Arm-based Dpsv5/Dpdsv5**: ~20% cheaper; multi-arch images required. - **Cluster Autoscaler** is the only scaler (no Karpenter equivalent); pool consolidation is more manual. ### Step 6: Safety Constraints The agent never proposes changes that risk data loss or sustained outage. Specifically: | Action | Hard Constraint | |--------|----------------| | Right-size memory | Never set request below historic working-set p99 | | Migrate to spot | Never for: databases, message brokers, stateful sets without proper PDB, single-replica deployments | | Shrink PV | Never in-place; always snapshot → smaller PV with cutover window | | Delete idle namespace | Only after 14-day notice + tag + archive (Helm uninstall preserves manifests) | | Migrate to Arm | Only after multi-arch image verified with `docker manifest inspect` | | Remove HPA | Never; only adjust | | Reduce replicas below PDB minAvailable | Never; adjust PDB first or keep replicas | ### Step 7: Implementation Plan (4 Weeks) ``` WEEK 1 — QUICK WINS (LOW RISK, HIGH ROI) Mon Run audit: collect kubectl + Prom + billing data Tue Identify C1/C2 candidates with > $500/mo savings each Wed Apply right-size patches to canary replicas Thu Monitor canary 48h Fri Roll out to full deployments; baseline new bill WEEK 2 — IDLE & UNUSED Mon Identify idle namespaces (C4) and idle deployments (C5) Tue Tag with `cost-review: pending-deletion`; notify owners Wed Scale-to-zero on confirmed idle workloads Thu Delete unused LBs (C7); delete oversized PVs (C6 staged) Fri Tally savings; update FinOps dashboard WEEK 3 — STRUCTURAL (NODE STRATEGY) Mon Add Karpenter (EKS) / Spot pool (GKE/AKS) Tue Add taints and tolerations on stateless workloads Wed Drain a small percentage to spot; observe interruption rate Thu Add Graviton/Arm pool; pin one workload as canary Fri Roll out to remaining stateless workloads WEEK 4 — COMMITMENTS & OBSERVABILITY Mon Compute baseline post-optimization Tue Purchase Savings Plan / CUDs at 70% of baseline Wed Tune logs/metrics ingestion (C15) Thu Add cost dashboards: per-namespace, per-team showback Fri Postmortem + savings report; schedule next quarter audit ``` ### Step 8: FinOps Reporting The agent produces a weekly / monthly report: ```markdown ## Cluster: prod-us-east-1 ### Savings achieved this month: $14,820 (-31%) | Category | $ saved | % of total | |----------|--------|-----------| | Right-size requests (C1+C2) | $6,420 | 43% | | Spot migration (C10) | $4,100 | 28% | | Graviton (C8) | $2,100 | 14% | | Logs ingestion (C15) | $1,900 | 13% | | LB cleanup (C7) | $300 | 2% | ### Remaining opportunities: $9,400/mo - HPA rollout pending on api-gateway: ~$850/mo - Savings Plan not yet purchased: ~$8,000/mo (commit 1-year) - Oversized PVs not yet migrated: ~$310/mo ### Risks accepted - Spot interruption rate: 3.2%/day (within budget of 5%) - Right-sized memory on cart-svc near p99 — monitoring OOMKills ``` ## Worked Examples ### Example 1: Right-Size A Java API **Input:** ``` Deployment: payments-api Replicas: 12 Container: payments resources.requests.cpu: 2 resources.requests.memory: 4Gi resources.limits.cpu: 2 resources.limits.memory: 4Gi Metrics (30d): cpu_usage p50: 0.18 p95: 0.31 p99: 0.52 memory_used p50: 1.8Gi p95: 2.1Gi p99: 2.4Gi Node pool: m5.2xlarge on-demand, ~$0.384/hr/instance ``` **Recommendation:** ```yaml resources: requests: cpu: 500m # was 2; p95 × 1.3 ≈ 0.4, floor 500m memory: 3Gi # was 4Gi; p99 × 1.25 = 3Gi limits: memory: 4Gi # keep memory limit at 1.5x request (3Gi → 4Gi) # cpu limit removed ``` **Savings:** (2 - 0.5 cpu) × $0.0384/cpu-hr × 24 × 30 × 12 replicas = ~$497/mo Plus memory: (4 - 3 GiB) × ~$0.0048/GiB-hr × 24 × 30 × 12 = ~$41/mo **Total: ~$540/mo just on this deployment.** ### Example 2: Spot Migration For A Stateless Fleet **Input:** EKS cluster, 60 nodes (m5.xlarge on-demand), 80% of workloads are stateless web/api. **Plan:** 1. Install Karpenter, add a NodePool with `karpenter.sh/capacity-type: spot`, instance types `m5,m5a,m6i,m6a,m7g.xlarge` (mixed for availability). 2. Keep an on-demand pool sized to the **stateful baseline** (databases, message queues, single-replica services). 3. Apply node affinity on stateless deployments: prefer spot, fallback on-demand. 4. Set PDBs `maxUnavailable: 25%` on every spot-eligible deployment. 5. Roll out workloads to spot in three waves (frontend → mid-tier → workers). 6. Monitor `karpenter_nodes_terminated{reason="interrupted"}` — interruption rate target < 5%/day. **Savings:** ~70% of compute on the stateless fleet × $0.192/hr m5.xlarge × 50 nodes × 24 × 30 = **~$4,840/mo savings**. ### Example 3: Idle Namespace Cleanup **Audit output:** ``` Namespace cpu_p95 mem_p95 age last_apply recommendation --------------- ------- -------- ---- ---------- -------------- demo-2024-q3 0.01 45Mi 9mo 8mo ago archive sandbox-alice 0.00 12Mi 6mo 5mo ago archive hackathon 0.02 180Mi 14mo 13mo ago archive ``` **Action:** tag, notify owners, archive after 14 days. Combined savings: full reclaim of ~6 nodes worth of allocation = ~$830/mo. ## Output The agent produces: - **Ranked recommendation list** — every C-type with $/month estimate, effort, risk - **Per-recommendation YAML patch** — kubectl-apply-ready or Helm-values diff - **Right-sizing report** — per-deployment requests/limits before/after - **Node strategy plan** — Karpenter / spot / Graviton migration steps - **Idle inventory** — namespaces, deployments, LBs, PVs flagged for cleanup - **FinOps dashboard spec** — Grafana / Datadog panel definitions for ongoing tracking - **4-week implementation plan** — daily checklist with safety gates - **Cloud-specific notes** — EKS/GKE/AKS particulars relevant to this cluster - **Pre-purchase commit plan** — Savings Plans / CUDs / Reservations sized to post-optimization baseline ## Common Scenarios ### "Our EKS bill jumped 40% this quarter" The agent diffs current cluster against a baseline 90 days ago: new namespaces, new deployments, replica creep, node pool growth, log ingestion delta. Most jumps are one-off: a new team, a stuck rollout, a logging misconfig. Identify and reverse, then run the standard audit. ### "We can't move to spot because we have stateful workloads" The agent splits the cluster: stateful pool stays on-demand (with reserved instances), stateless moves to spot. The mistake to avoid is one node pool serving both; taints and tolerations are mandatory. ### "Our requests look right but the bill is still high" Probably nodes, not pods. Look at node utilization (`kubectl top nodes`); if average node utilization is below 50%, the cluster is over-provisioned at the node layer (binpacking failure or large pod requests blocking dense placement). Karpenter / cluster-autoscaler tuning beats per-pod right-sizing here. ### "Should I use VPA in auto mode?" No, in most production clusters. VPA in auto mode restarts pods when it adjusts requests — that's a disruption budget hit on every change. Use `recommendationOnly` and apply manually, or use VPA with a long `targetRef` change budget. ### "Kubecost says one thing, AWS Cost Explorer says another" Kubecost allocates *cluster cost to namespaces*; AWS bills *resources*. The deltas: cluster control-plane cost (in AWS but not Kubecost by default), cross-region egress, registry transfer, NAT gateway. Reconcile both before claiming savings. ## Tips For Best Results - Provide at least 30 days of metrics — anything shorter misses the weekend pattern and over-rightsizes - Share both `kubectl get all -A -o json` and a Prometheus snapshot — neither alone is enough - State the cloud provider (EKS / GKE / AKS) and region — pricing varies materially - Identify any workloads under SLAs or compliance constraints — those get conservative sizing margins - Include the previous quarter's bill — flat-rate analysis misses growth-driven savings - Mention any committed-spend agreements (EDP, GCP commitment, Azure MCA) — recommendations adjust to maximize commit utilization - If running Kubecost or OpenCost, include its allocation export — the agent uses it directly instead of recomputing ## When NOT To Use - **Brand-new cluster (< 30 days old)** — not enough data; metrics will mislead. Wait until the cluster has had a full month including a release cycle. - **Single-tenant cluster running one workload at near 100% utilization** — already optimized; further savings are at the cloud-commit layer, not Kubernetes. - **Non-Kubernetes workloads (ECS, Cloud Run, App Service)** — different scaling primitives and cost models; use a serverless cost optimizer instead. - **Cluster with strict regulatory pinning** (FedRAMP, PCI-DSS regions, sovereign cloud) — instance type and region freedom is constrained; many recommendations don't apply. - **Pre-production performance test environments** — these are intentionally over-provisioned; right-sizing them invalidates load-test results. - **Clusters where the cost is dwarfed by other line items** (e.g. $2k/mo K8s vs $50k/mo data warehouse) — optimize the bigger line first.


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: clean (LOW)