Skill: briancolinger/aws-ecs-monitor (version 1.0.1)

--- name: aws-ecs-monitor version: 1.0.1 description: AWS ECS production health monitoring with CloudWatch log analysis — monitors ECS service health, ALB targets, SSL certificates, and provides deep CloudWatch log analysis for error categorization, restart detection, and production alerts. metadata: openclaw: requires: bins: ["aws", "curl", "python3"] anyBins: ["openssl"] --- # AWS ECS Monitor Production health monitoring and log analysis for AWS ECS services. ## What It Does - **Health Checks**: HTTP probes against your domain, ECS service status (desired vs running), ALB target group health, SSL certificate expiry - **Log Analysis**: Pulls CloudWatch logs, categorizes errors (panics, fatals, OOM, timeouts, 5xx), detects container restarts, filters health check noise - **Auto-Diagnosis**: Reads health status and automatically investigates failing services via log analysis ## Prerequisites - `aws` CLI configured with appropriate IAM permissions: - `ecs:ListServices`, `ecs:DescribeServices` - `elasticloadbalancing:DescribeTargetGroups`, `elasticloadbalancing:DescribeTargetHealth` - `logs:FilterLogEvents`, `logs:DescribeLogGroups` - `curl` for HTTP health checks - `python3` for JSON processing and log analysis - `openssl` for SSL certificate checks (optional) ## Configuration All configuration is via environment variables: | Variable | Required | Default | Description | |---|---|---|---| | `ECS_CLUSTER` | **Yes** | — | ECS cluster name | | `ECS_REGION` | No | `us-east-1` | AWS region | | `ECS_DOMAIN` | No | — | Domain for HTTP/SSL checks (skip if unset) | | `ECS_SERVICES` | No | auto-detect | Comma-separated service names to monitor | | `ECS_HEALTH_STATE` | No | `./data/ecs-health.json` | Path to write health state JSON | | `ECS_HEALTH_OUTDIR` | No | `./data/` | Output directory for logs and alerts | | `ECS_LOG_PATTERN` | No | `/ecs/{service}` | CloudWatch log group pattern (`{service}` is replaced) | | `ECS_HTTP_ENDPOINTS` | No | — | Comma-separated `name=url` pairs for HTTP probes | ## Directories Written - **`ECS_HEALTH_STATE`** (default: `./data/ecs-health.json`) — Health state JSON file - **`ECS_HEALTH_OUTDIR`** (default: `./data/`) — Output directory for logs, alerts, and analysis reports ## Scripts ### `scripts/ecs-health.sh` — Health Monitor ```bash # Full check ECS_CLUSTER=my-cluster ECS_DOMAIN=example.com ./scripts/ecs-health.sh # JSON output only ECS_CLUSTER=my-cluster ./scripts/ecs-health.sh --json # Quiet mode (no alerts, just status file) ECS_CLUSTER=my-cluster ./scripts/ecs-health.sh --quiet ``` Exit codes: `0` = healthy, `1` = unhealthy/degraded, `2` = script error ### `scripts/cloudwatch-logs.sh` — Log Analyzer ```bash # Pull raw logs from a service ECS_CLUSTER=my-cluster ./scripts/cloudwatch-logs.sh pull my-api --minutes 30 # Show errors across all services ECS_CLUSTER=my-cluster ./scripts/cloudwatch-logs.sh errors all --minutes 120 # Deep analysis with error categorization ECS_CLUSTER=my-cluster ./scripts/cloudwatch-logs.sh diagnose --minutes 60 # Detect container restarts ECS_CLUSTER=my-cluster ./scripts/cloudwatch-logs.sh restarts my-api # Auto-diagnose from health state file ECS_CLUSTER=my-cluster ./scripts/cloudwatch-logs.sh auto-diagnose # Summary across all services ECS_CLUSTER=my-cluster ./scripts/cloudwatch-logs.sh summary --minutes 120 ``` Options: `--minutes N` (default: 60), `--json`, `--limit N` (default: 200), `--verbose` ## Auto-Detection When `ECS_SERVICES` is not set, both scripts auto-detect services from the cluster: ```bash aws ecs list-services --cluster $ECS_CLUSTER ``` Log groups are resolved by pattern (default `/ecs/{service}`). Override with `ECS_LOG_PATTERN`: ```bash # If your log groups are /ecs/prod/my-api, /ecs/prod/my-frontend, etc. ECS_LOG_PATTERN="/ecs/prod/{service}" ECS_CLUSTER=my-cluster ./scripts/cloudwatch-logs.sh diagnose ``` ## Integration The health monitor can trigger the log analyzer for auto-diagnosis when issues are detected. Set `ECS_HEALTH_OUTDIR` to a shared directory and run both scripts together: ```bash export ECS_CLUSTER=my-cluster export ECS_DOMAIN=example.com export ECS_HEALTH_OUTDIR=./data # Run health check (auto-triggers log analysis on failure) ./scripts/ecs-health.sh # Or run log analysis independently ./scripts/cloudwatch-logs.sh auto-diagnose --minutes 30 ``` ## Error Categories The log analyzer classifies errors into: - `panic` — Go panics - `fatal` — Fatal errors - `oom` — Out of memory - `timeout` — Connection/request timeouts - `connection_error` — Connection refused/reset - `http_5xx` — HTTP 500-level responses - `python_traceback` — Python tracebacks - `exception` — Generic exceptions - `auth_error` — Permission/authorization failures - `structured_error` — JSON-structured error logs - `error` — Generic ERROR-level messages Health check noise (GET/HEAD `/health` from ALB) is automatically filtered from error counts and HTTP status distribution.

path: scripts/cloudwatch-logs.sh
content: #!/usr/bin/env bash
# =============================================================================
# ECS CloudWatch Log Puller & Analyzer
#
# Pull and analyze CloudWatch logs from ECS services.
# Integrates with ecs-health.sh for auto-diagnosis.
#
# Configuration (environment variables):
#   ECS_CLUSTER        (required) ECS cluster name
#   ECS_REGION         AWS region (default: us-east-1)
#   ECS_SERVICES       Comma-separated service names (auto-detect if unset)
#   ECS_HEALTH_STATE   Health state file path (default: ./data/ecs-health.json)
#   ECS_HEALTH_OUTDIR  Output directory (default: ./data/)
#   ECS_LOG_PATTERN    Log group pattern (default: /ecs/{service})
#
# Usage:
#   ./scripts/cloudwatch-logs.sh pull [service] [--minutes N] [--filter PATTERN]
#   ./scripts/cloudwatch-logs.sh errors [service] [--minutes N]
#   ./scripts/cloudwatch-logs.sh diagnose [service] [--minutes N]
#   ./scripts/cloudwatch-logs.sh restarts [service] [--minutes N]
#   ./scripts/cloudwatch-logs.sh summary [--minutes N]
#   ./scripts/cloudwatch-logs.sh auto-diagnose [--minutes N]
#
# Exit codes: 0 = success, 1 = issues found, 2 = script error
# =============================================================================
set -uo pipefail

# ---------------------------------------------------------------------------
# Configuration
# ---------------------------------------------------------------------------
if [[ -z "${ECS_CLUSTER:-}" ]]; then
  echo "ERROR: ECS_CLUSTER environment variable is required" >&2
  exit 2
fi

CLUSTER="${ECS_CLUSTER}"
REGION="${ECS_REGION:-us-east-1}"
OUTDIR="${ECS_HEALTH_OUTDIR:-./data}"
DATA_DIR="${OUTDIR}/logs"
HEALTH_FILE="${ECS_HEALTH_STATE:-${OUTDIR}/ecs-health.json}"
LOG_PATTERN="${ECS_LOG_PATTERN:-/ecs/{service}}"

COMMAND="" ; SERVICE="" ; MINUTES=60 ; FILTER=""
JSON_OUT=false ; LIMIT=200 ; VERBOSE=false

# ---------------------------------------------------------------------------
# Auto-detect services from cluster
# ---------------------------------------------------------------------------
detect_services() {
  aws ecs list-services --cluster "$CLUSTER" --region "$REGION" --output json 2>/dev/null \
    | python3 -c "
import json, sys
data = json.load(sys.stdin)
print(' '.join(a.split('/')[-1] for a in data.get('serviceArns', [])))
" 2>/dev/null
}

declare -a ALL_SERVICES_ARR
if [[ -n "${ECS_SERVICES:-}" ]]; then
  IFS=',' read -ra ALL_SERVICES_ARR <<< "$ECS_SERVICES"
else
  read -ra ALL_SERVICES_ARR <<< "$(detect_services)"
fi

if [[ ${#ALL_SERVICES_ARR[@]} -eq 0 ]]; then
  echo "ERROR: No services found in cluster '$CLUSTER'." >&2
  exit 2
fi

ALL_SERVICES="${ALL_SERVICES_ARR[*]}"

# Build log group mapping
declare -A LOG_GROUPS
for svc in ${ALL_SERVICES}; do
  LOG_GROUPS[$svc]="${LOG_PATTERN//\{service\}/$svc}"
done

# ---------------------------------------------------------------------------
# Parse arguments
# ---------------------------------------------------------------------------
parse_args() {
  COMMAND="${1:-}" ; shift || true
  while [[ $# -gt 0 ]]; do
    case "$1" in
      --minutes)  MINUTES="${2:-60}"; shift 2 ;;
      --filter)   FILTER="${2:-}"; shift 2 ;;
      --json)     JSON_OUT=true; shift ;;
      --limit)    LIMIT="${2:-200}"; shift 2 ;;
      --verbose)  VERBOSE=true; shift ;;
      -*)         echo "Unknown option: $1" >&2; exit 2 ;;
      *)          [[ -z "$SERVICE" ]] && SERVICE="$1"; shift ;;
    esac
  done
  [[ -z "$SERVICE" ]] && SERVICE="all"
}

resolve_services() {
  local svc="$1"
  if [[ "$svc" == "all" ]]; then echo "$ALL_SERVICES"; return; fi
  for s in ${ALL_SERVICES}; do
    [[ "$s" == "$svc" ]] && echo "$svc" && return
  done
  echo "Unknown service: $svc (available: $ALL_SERVICES)" >&2; exit 2
}

# ---------------------------------------------------------------------------
# Pull logs from a single service
# ---------------------------------------------------------------------------
pull_service_logs() {
  local svc="$1" log_group="${LOG_GROUPS[$1]}"
  local start_ms=$(( ($(date +%s) - MINUTES * 60) * 1000 ))
  local end_ms=$(( $(date +%s) * 1000 ))
  local cmd=(aws logs filter-log-events --log-group-name "$log_group"
    --region "$REGION" --start-time "$start_ms" --end-time "$end_ms"
    --limit "$LIMIT" --output json)
  [[ -n "$FILTER" ]] && cmd+=(--filter-pattern "$FILTER")
  "${cmd[@]}" 2>/dev/null || echo '{"events":[]}'
}

# ---------------------------------------------------------------------------
# cmd: pull
# ---------------------------------------------------------------------------
cmd_pull() {
  for svc in $(resolve_services "$SERVICE"); do
    local result; result=$(pull_service_logs "$svc")
    local lg="${LOG_GROUPS[$svc]}"
    if $JSON_OUT; then
      echo "$result" | python3 -c "
import json,sys; d=json.load(sys.stdin); e=d.get('events',[])
print(json.dumps({'service':'$svc','log_group':'$lg','count':len(e),'events':e},indent=2))"
    else
      local count; count=$(echo "$result" | python3 -c "import json,sys;print(len(json.load(sys.stdin).get('events',[])))" 2>/dev/null)
      echo "=== $svc ($lg) — $count events in last ${MINUTES}m ==="
      echo "$result" | python3 -c "
import json,sys,datetime
for e in json.load(sys.stdin).get('events',[]):
    ts=datetime.datetime.fromtimestamp(e['timestamp']/1000,datetime.timezone.utc).strftime('%H:%M:%S')
    m=e.get('message','').strip(); m=m[:500]+'...' if len(m)>500 else m
    print(f'  [{ts}] {m}')" 2>/dev/null
      echo ""
    fi
  done
}

# ---------------------------------------------------------------------------
# cmd: errors
# ---------------------------------------------------------------------------
cmd_errors() {
  local orig_filter="$FILTER" found=false
  FILTER='?ERROR ?error ?panic ?fatal ?FATAL ?"level":"error" ?Traceback ?Exception'
  for svc in $(resolve_services "$SERVICE"); do
    local result; result=$(pull_service_logs "$svc")
    local count; count=$(echo "$result" | python3 -c "import json,sys;print(len(json.load(sys.stdin).get('events',[])))" 2>/dev/null)
    if [[ "$count" -gt 0 ]]; then
      found=true
      if $JSON_OUT; then
        echo "$result" | python3 -c "
import json,sys; d=json.load(sys.stdin); e=d.get('events',[])
print(json.dumps({'service':'$svc','error_count':len(e),'events':e},indent=2))"
      else
        echo "🔴 $svc — $count error(s) in last ${MINUTES}m:"
        echo "$result" | python3 -c "
import json,sys,datetime
for e in json.load(sys.stdin).get('events',[]):
    ts=datetime.datetime.fromtimestamp(e['timestamp']/1000,datetime.timezone.utc).strftime('%H:%M:%S')
    m=e.get('message','').strip(); m=m[:300]+'...' if len(m)>300 else m
    print(f'  [{ts}] {m}')" 2>/dev/null
        echo ""
      fi
    else
      $JSON_OUT || echo "✅ $svc — no errors in last ${MINUTES}m"
    fi
  done
  FILTER="$orig_filter"
  [[ "$found" == "true" ]] && return 1 || return 0
}

# ---------------------------------------------------------------------------
# cmd: restarts
# ---------------------------------------------------------------------------
cmd_restarts() {
  local orig_filter="$FILTER"
  FILTER='?"Shutting down" ?"Starting" ?"started" ?"Uvicorn running" ?"Server started" ?"next start" ?"listening" ?"gracefully" ?"SIGTERM" ?"signal"'
  for svc in $(resolve_services "$SERVICE"); do
    pull_service_logs "$svc" | python3 -c "
import json,sys,datetime,re
events=json.load(sys.stdin).get('events',[])
up_re=[r'(?i)(starting|started|listening|running on|ready)',r'(?i)(uvicorn running|next start|gin-gonic)']
dn_re=[r'(?i)(shutting down|stopped|sigterm|graceful)']
ups,dns=[],[]
for e in events:
    m,ts=e.get('message','').strip(),datetime.datetime.fromtimestamp(e['timestamp']/1000,datetime.timezone.utc).strftime('%Y-%m-%d %H:%M:%S')
    if any(re.search(p,m) for p in up_re): ups.append((ts,m[:200]))
    if any(re.search(p,m) for p in dn_re): dns.append((ts,m[:200]))
print(f'=== $svc — {len(ups)} startups, {len(dns)} shutdowns (~{min(len(ups),len(dns))} restarts) in last ${MINUTES}m ===')
for ts,k,m in sorted([(t,'START',m) for t,m in ups]+[(t,'STOP',m) for t,m in dns]):
    print(f\"  {'🟢' if k=='START' else '🔴'} [{ts}] {m}\")
print()" 2>/dev/null
  done
  FILTER="$orig_filter"
}

# ---------------------------------------------------------------------------
# cmd: diagnose — deep analysis
# ---------------------------------------------------------------------------
cmd_diagnose() {
  mkdir -p "$DATA_DIR"
  local lg_json="{"
  local first=true
  for svc in $(resolve_services "$SERVICE"); do
    $first || lg_json+=","
    first=false
    lg_json+="\"$svc\":\"${LOG_GROUPS[$svc]}\""
  done
  lg_json+="}"

  CW_SERVICES="$(resolve_services "$SERVICE")" CW_REGION="$REGION" \
  CW_MINUTES="$MINUTES" CW_LIMIT="$LIMIT" CW_JSON_OUT="$JSON_OUT" \
  CW_VERBOSE="$VERBOSE" CW_LOG_GROUPS="$lg_json" python3 <<'PYEOF'
import json, sys, subprocess, re, datetime, os
from collections import Counter

services = os.environ["CW_SERVICES"].split()
region, minutes, limit = os.environ["CW_REGION"], int(os.environ["CW_MINUTES"]), int(os.environ["CW_LIMIT"])
json_out, verbose = os.environ["CW_JSON_OUT"]=="true", os.environ["CW_VERBOSE"]=="true"
log_groups = json.loads(os.environ["CW_LOG_GROUPS"])

ERROR_PATS = [
    (r'"level"\s*:\s*"error"',"structured_error"),(r'(?i)\bpanic\b',"panic"),
    (r'(?i)\bfatal\b',"fatal"),(r'(?i)\bERROR\b',"error"),
    (r'Traceback \(most recent',"python_traceback"),(r'(?i)exception',"exception"),
    (r'\| 5\d{2} \|',"http_5xx"),(r'(?i)timeout|timed out',"timeout"),
    (r'(?i)connection refused|connection reset',"connection_error"),
    (r'(?i)out of memory|OOM',"oom"),(r'(?i)permission denied|unauthorized|forbidden',"auth_error"),
]
RESTART_PATS = [(r'(?i)shutting down|stopped|sigterm',"shutdown"),
                (r'(?i)starting|started|listening|running on|ready',"startup")]
HTTP_RE = re.compile(r'\[GIN\]\s+\S+\s+-\s+\S+\s+\|\s+(\d{3})\s+\|')
HEALTH_RE = re.compile(r'(?:HEAD|GET)\s+"/health"')

_now = datetime.datetime.now(datetime.timezone.utc)
start_ms = int((_now - datetime.timedelta(minutes=minutes)).timestamp() * 1000)
end_ms = int(_now.timestamp() * 1000)
ts_fmt = lambda ms: datetime.datetime.fromtimestamp(ms/1000, datetime.timezone.utc).strftime("%H:%M:%S")
all_findings = {}

for svc in services:
    lg = log_groups.get(svc)
    if not lg: continue
    try:
        r = subprocess.run(["aws","logs","filter-log-events","--log-group-name",lg,
            "--region",region,"--start-time",str(start_ms),"--end-time",str(end_ms),
            "--limit",str(limit),"--output","json"], capture_output=True, text=True, timeout=30)
        events = json.loads(r.stdout).get("events", [])
    except Exception as e:
        all_findings[svc] = {"status":"error","error":str(e)}; continue
    if not events:
        all_findings[svc] = {"status":"no_logs","event_count":0}; continue

    f = {"event_count":len(events),"errors":[],"error_categories":Counter(),
         "restarts":[],"http_status_codes":Counter(),"unique_streams":set(),
         "time_range":{"start":ts_fmt(events[0]["timestamp"]),"end":ts_fmt(events[-1]["timestamp"])}}

    for evt in events:
        msg, ts, stream = evt.get("message","").strip(), evt.get("timestamp",0), evt.get("logStreamName","")
        f["unique_streams"].add(stream)
        is_hc = bool(HEALTH_RE.search(msg))
        if not is_hc:
            for pat, cat in ERROR_PATS:
                if re.search(pat, msg):
                    f["error_categories"][cat] += 1
                    f["errors"].append({"timestamp":ts_fmt(ts),"category":cat,"message":msg[:300],"stream":stream[-12:]})
                    break
        for pat, kind in RESTART_PATS:
            if re.search(pat, msg):
                f["restarts"].append({"timestamp":ts_fmt(ts),"kind":kind,"message":msg[:200]}); break
        hm = HTTP_RE.search(msg)
        if hm:
            code = hm.group(1)
            if not (code == "404" and is_hc): f["http_status_codes"][code] += 1

    ec = sum(f["error_categories"].values())
    rc = sum(1 for r in f["restarts"] if r["kind"] == "shutdown")
    ns = len(f["unique_streams"])
    nr = max(2, ns * max(1, minutes // 30))
    if any(c in f["error_categories"] for c in ("panic","fatal","oom")): f["status"] = "critical"
    elif ec > 10: f["status"] = "unhealthy"
    elif ec > 0 or rc > nr: f["status"] = "degraded"
    else: f["status"] = "healthy"

    f["unique_streams"] = ns
    f["error_categories"] = dict(f["error_categories"])
    f["http_status_codes"] = dict(f["http_status_codes"])
    seen, uniq = set(), []
    for err in f["errors"]:
        k = (err["category"], err["message"][:80])
        if k not in seen: seen.add(k); uniq.append(err)
    f["unique_error_count"], f["total_error_count"] = len(uniq), len(f["errors"])
    f["errors"] = uniq[:20] if not verbose else f["errors"][:50]
    all_findings[svc] = f

if json_out:
    print(json.dumps(all_findings, indent=2, default=str))
else:
    has_issues = False
    for svc, f in sorted(all_findings.items()):
        st = f.get("status","unknown")
        ic = {"healthy":"✅","degraded":"⚠️","unhealthy":"🔴","critical":"🚨","no_logs":"⚪","error":"❌"}.get(st,"❓")
        print(f"{ic} {svc} — {st} ({f.get('event_count',0)} events, {f.get('unique_streams',0)} streams)")
        if st in ("degraded","unhealthy","critical"): has_issues = True
        cats = f.get("error_categories",{})
        if cats:
            print(f"  Errors: {f.get('total_error_count',0)} total ({', '.join(f'{k}:{v}' for k,v in sorted(cats.items(),key=lambda x:-x[1]))})")
        http = f.get("http_status_codes",{})
        non200 = {k:v for k,v in http.items() if k != "200"}
        if non200:
            print(f"  HTTP: {sum(http.values())} total, non-200: {', '.join(f'{k}:{v}' for k,v in sorted(non200.items()))}")
        rs = f.get("restarts",[])
        sd, su = sum(1 for r in rs if r["kind"]=="shutdown"), sum(1 for r in rs if r["kind"]=="startup")
        if sd or su: print(f"  Restarts: {sd} shutdowns, {su} startups")
        errs = f.get("errors",[])
        if errs:
            print(f"  Sample errors ({min(len(errs),5)} of {f.get('unique_error_count',len(errs))} unique):")
            for e in errs[:5]:
                m = e["message"][:120]+"..." if len(e["message"])>120 else e["message"]
                print(f"    [{e['timestamp']}] ({e['category']}) {m}")
        print()
    if has_issues: sys.exit(1)
PYEOF
}

# ---------------------------------------------------------------------------
# cmd: summary
# ---------------------------------------------------------------------------
cmd_summary() { SERVICE="all"; cmd_diagnose; }

# ---------------------------------------------------------------------------
# cmd: auto-diagnose — parse health state, then run diagnose on failing svcs
# ---------------------------------------------------------------------------
cmd_auto_diagnose() {
  if [[ ! -f "$HEALTH_FILE" ]]; then
    echo "No health status file found at $HEALTH_FILE — run ecs-health.sh first."
    exit 2
  fi

  # Parse health file to find failing services and print header
  local failing_svcs
  failing_svcs=$(python3 -c "
import json, sys
with open('$HEALTH_FILE') as f: health = json.load(f)
overall = health.get('overall','unknown')
checks = health.get('checks',{})
print(f'Production Status: {overall} (from {health.get(\"timestamp\",\"unknown\")})', file=sys.stderr)
print(file=sys.stderr)

svcs, issues = [], []
ecs = checks.get('ecs_services',{})
if ecs.get('status') not in ('healthy',):
    for n, d in ecs.get('services',{}).items():
        h = d.get('health','unknown')
        if h in ('degraded','unhealthy','failed'):
            svcs.append(n)
            issues.append(f'{n}: {h} (desired={d.get(\"desired\")}, running={d.get(\"running\")})')
for key in ('domain_http',):
    c = checks.get(key,{})
    if c.get('status') != 'healthy':
        issues.append(f'{key}: {c.get(\"status\")} (code={c.get(\"http_code\")})')
for name, c in checks.items():
    if name.startswith('http_') and isinstance(c,dict) and c.get('status') != 'healthy':
        issues.append(f'{name}: {c.get(\"status\")}')
alb = checks.get('alb_targets',{})
if alb.get('status') not in ('healthy',):
    for tn, td in alb.get('target_groups',{}).items():
        if td.get('status') not in ('healthy','empty'):
            issues.append(f'ALB {tn}: {td.get(\"status\")}')

if not svcs and overall == 'healthy':
    print('', file=sys.stderr)
    print('✅ All services healthy — no log analysis needed.', file=sys.stderr)
    sys.exit(0)

if issues:
    print('Issues detected:', file=sys.stderr)
    for i in issues: print(f'  ⚠️  {i}', file=sys.stderr)
    print(file=sys.stderr)

# If we couldn't identify specific services, check those with desired > 0
if not svcs:
    ecs_svcs = ecs.get('services',{})
    svcs = [n for n,d in ecs_svcs.items() if d.get('desired',0) > 0]

# Output service names to stdout for bash to consume
print(' '.join(svcs) if svcs else '')
" 2>&1 1>/tmp/ecs-ad-svcs.$$)

  # Print the header/issues (stderr was redirected to stdout above)
  echo "$failing_svcs"

  # Read the services list
  local svc_list
  svc_list=$(cat /tmp/ecs-ad-svcs.$$ 2>/dev/null)
  rm -f /tmp/ecs-ad-svcs.$$

  if [[ -z "$svc_list" ]]; then
    return 0
  fi

  echo "Analyzing logs for: $svc_list (last ${MINUTES}m)"
  echo "$(printf '=%.0s' {1..60})"
  echo ""

  # Diagnose each failing service using the existing diagnose command
  for svc in $svc_list; do
    # Check if this service is in our known list
    local known=false
    for s in ${ALL_SERVICES}; do
      [[ "$s" == "$svc" ]] && known=true && break
    done
    if $known; then
      SERVICE="$svc"
      LIMIT=300
      cmd_diagnose
    else
      echo "⚠️  $svc — not in detected service list, skipping"
    fi
  done
}

# ---------------------------------------------------------------------------
# Usage
# ---------------------------------------------------------------------------
usage() {
  cat <<EOF
ECS CloudWatch Log Puller & Analyzer

Usage:
  $0 pull [service] [--minutes N] [--filter PAT]   Pull raw logs
  $0 errors [service] [--minutes N]                  Show errors only
  $0 diagnose [service] [--minutes N]                Deep log analysis
  $0 restarts [service] [--minutes N]                Detect restarts
  $0 summary [--minutes N]                           Summary across all
  $0 auto-diagnose [--minutes N]                     Health state + analyze

Services: auto-detected from cluster (or set ECS_SERVICES). "all" = default.
Options:  --minutes N (60), --json, --limit N (200), --verbose

Environment:
  ECS_CLUSTER       (required) ECS cluster name
  ECS_REGION        AWS region (default: us-east-1)
  ECS_SERVICES      Comma-separated service names
  ECS_HEALTH_STATE  Health state file path
  ECS_HEALTH_OUTDIR Output directory
  ECS_LOG_PATTERN   Log group pattern (default: /ecs/{service})
EOF
}

# ---------------------------------------------------------------------------
# Main
# ---------------------------------------------------------------------------
parse_args "$@"
mkdir -p "$DATA_DIR"
case "$COMMAND" in
  pull)           cmd_pull ;;
  errors)         cmd_errors ;;
  diagnose)       cmd_diagnose ;;
  restarts)       cmd_restarts ;;
  summary)        cmd_summary ;;
  auto-diagnose)  cmd_auto_diagnose ;;
  help|-h|--help) usage ;;
  *)              echo "Unknown command: ${COMMAND:-<none>}" >&2; usage >&2; exit 2 ;;
esac


path: scripts/ecs-health.sh
content: #!/usr/bin/env bash
# =============================================================================
# ECS Production Health Monitor
#
# Generic health monitor for AWS ECS services. Checks:
#   1. Domain HTTP probe (if ECS_DOMAIN is set)
#   2. Custom HTTP endpoint probes (if ECS_HTTP_ENDPOINTS is set)
#   3. ECS service status (desired vs running task counts)
#   4. ALB target group health
#   5. SSL certificate expiry (if ECS_DOMAIN is set)
#
# Configuration (environment variables):
#   ECS_CLUSTER        (required) ECS cluster name
#   ECS_REGION         AWS region (default: us-east-1)
#   ECS_DOMAIN         Domain for HTTP/SSL checks (optional)
#   ECS_SERVICES       Comma-separated service names (auto-detect if unset)
#   ECS_HEALTH_STATE   State file path (default: ./data/ecs-health.json)
#   ECS_HEALTH_OUTDIR  Output directory (default: ./data/)
#   ECS_HTTP_ENDPOINTS Comma-separated name=url pairs for HTTP probes (optional)
#
# Usage:
#   ./scripts/ecs-health.sh              # Full check
#   ./scripts/ecs-health.sh --quiet      # No alerts, just status file
#   ./scripts/ecs-health.sh --json       # JSON output to stdout
#
# Exit codes:
#   0 = all healthy
#   1 = one or more checks failed
#   2 = script error (missing deps, missing config)
# =============================================================================
set -uo pipefail

# ---------------------------------------------------------------------------
# Configuration
# ---------------------------------------------------------------------------
if [[ -z "${ECS_CLUSTER:-}" ]]; then
  echo "ERROR: ECS_CLUSTER environment variable is required" >&2
  exit 2
fi

CLUSTER="${ECS_CLUSTER}"
REGION="${ECS_REGION:-us-east-1}"
DOMAIN="${ECS_DOMAIN:-}"
OUTDIR="${ECS_HEALTH_OUTDIR:-./data}"
STATUS_FILE="${ECS_HEALTH_STATE:-${OUTDIR}/ecs-health.json}"

QUIET=false
JSON_ONLY=false
for arg in "$@"; do
  case "$arg" in
    --quiet) QUIET=true ;;
    --json)  JSON_ONLY=true ;;
  esac
done

# Ensure output directory exists
mkdir -p "$(dirname "$STATUS_FILE")"
mkdir -p "$OUTDIR"

# Dependencies check
for cmd in curl aws python3; do
  if ! command -v "$cmd" &>/dev/null; then
    echo "Missing dependency: $cmd" >&2
    exit 2
  fi
done

# ---------------------------------------------------------------------------
# Auto-detect services from cluster
# ---------------------------------------------------------------------------
detect_services() {
  local svc_arns
  svc_arns=$(aws ecs list-services --cluster "$CLUSTER" --region "$REGION" --output json 2>/dev/null)
  if [[ -z "$svc_arns" ]]; then
    echo "" 
    return
  fi
  echo "$svc_arns" | python3 -c "
import json, sys
data = json.load(sys.stdin)
arns = data.get('serviceArns', [])
names = [a.split('/')[-1] for a in arns]
print(' '.join(names))
" 2>/dev/null
}

if [[ -n "${ECS_SERVICES:-}" ]]; then
  # Convert comma-separated to array
  IFS=',' read -ra SERVICES <<< "$ECS_SERVICES"
else
  read -ra SERVICES <<< "$(detect_services)"
fi

if [[ ${#SERVICES[@]} -eq 0 ]]; then
  echo "ERROR: No services found in cluster '$CLUSTER'. Set ECS_SERVICES or check cluster name." >&2
  exit 2
fi

# Timestamp
NOW=$(date +%s)
NOW_ISO=$(date -u +"%Y-%m-%dT%H:%M:%SZ")

# Collect results into temp directory
TMPDIR=$(mktemp -d)
trap 'rm -rf "$TMPDIR"' EXIT
OVERALL="healthy"

# ---------------------------------------------------------------------------
# 1. Domain HTTP Check (if ECS_DOMAIN is set)
# ---------------------------------------------------------------------------
if [[ -n "$DOMAIN" ]]; then
  fe_status=$(curl -s -o /dev/null -w "%{http_code}" --max-time 10 "https://$DOMAIN/" 2>/dev/null || echo "000")
  fe_latency=$(curl -s -o /dev/null -w "%{time_total}" --max-time 10 "https://$DOMAIN/" 2>/dev/null || echo "0")

  if [[ "$fe_status" == "200" ]]; then
    echo '{"status":"healthy","http_code":'"$fe_status"',"latency_s":'"$fe_latency"'}' > "$TMPDIR/domain_http.json"
  else
    echo '{"status":"unhealthy","http_code":'"$fe_status"',"latency_s":'"$fe_latency"',"error":"Expected 200, got '"$fe_status"'"}' > "$TMPDIR/domain_http.json"
    OVERALL="unhealthy"
  fi
fi

# ---------------------------------------------------------------------------
# 2. Custom HTTP Endpoint Probes (if ECS_HTTP_ENDPOINTS is set)
# ---------------------------------------------------------------------------
if [[ -n "${ECS_HTTP_ENDPOINTS:-}" ]]; then
  IFS=',' read -ra ENDPOINTS <<< "$ECS_HTTP_ENDPOINTS"
  for ep in "${ENDPOINTS[@]}"; do
    ep_name="${ep%%=*}"
    ep_url="${ep#*=}"
    ep_status=$(curl -s -o /dev/null -w "%{http_code}" --max-time 10 "$ep_url" 2>/dev/null || echo "000")
    ep_latency=$(curl -s -o /dev/null -w "%{time_total}" --max-time 10 "$ep_url" 2>/dev/null || echo "0")

    # Any non-zero HTTP response means reachable
    if [[ "$ep_status" != "000" && "$ep_status" != "" ]]; then
      echo '{"status":"healthy","http_code":'"$ep_status"',"latency_s":'"$ep_latency"',"url":"'"$ep_url"'"}' > "$TMPDIR/http_${ep_name}.json"
    else
      echo '{"status":"unhealthy","http_code":'"$ep_status"',"latency_s":'"$ep_latency"',"url":"'"$ep_url"'","error":"Unreachable"}' > "$TMPDIR/http_${ep_name}.json"
      OVERALL="unhealthy"
    fi
  done
fi

# ---------------------------------------------------------------------------
# 3. ECS Service Status
# ---------------------------------------------------------------------------
ecs_json=$(aws ecs describe-services \
  --cluster "$CLUSTER" \
  --services "${SERVICES[@]}" \
  --region "$REGION" \
  --output json 2>/dev/null)

if [[ -z "$ecs_json" ]]; then
  echo '{"status":"error","error":"Failed to query ECS"}' > "$TMPDIR/ecs_services.json"
  OVERALL="unhealthy"
else
  ecs_result=$(echo "$ecs_json" | python3 -c "
import json, sys

data = json.load(sys.stdin)
services = data.get('services', [])
failures = data.get('failures', [])

result = {'status': 'healthy', 'services': {}}
issues = []

for svc in services:
    name = svc['serviceName']
    desired = svc['desiredCount']
    running = svc['runningCount']
    status = svc['status']

    svc_health = 'healthy'

    if desired == 0:
        # Scaled down — not an error, just informational
        if status != 'ACTIVE':
            svc_health = 'unhealthy'
            issues.append(f'{name}: status={status}')
        else:
            svc_health = 'scaled_down'
    elif running < desired:
        svc_health = 'degraded'
        issues.append(f'{name}: {running}/{desired} tasks running')
    elif status != 'ACTIVE':
        svc_health = 'unhealthy'
        issues.append(f'{name}: status={status}')

    result['services'][name] = {
        'desired': desired,
        'running': running,
        'status': status,
        'health': svc_health
    }

for f in failures:
    arn = f.get('arn', 'unknown')
    reason = f.get('reason', 'unknown')
    name = arn.split('/')[-1] if '/' in arn else arn
    issues.append(f'{name}: FAILURE - {reason}')
    result['services'][name] = {'health': 'failed', 'error': reason}

if issues:
    result['status'] = 'degraded'
    result['issues'] = issues

print(json.dumps(result))
" 2>/dev/null)

  if [[ -n "$ecs_result" ]]; then
    echo "$ecs_result" > "$TMPDIR/ecs_services.json"
  else
    echo '{"status":"error","error":"Failed to parse ECS response"}' > "$TMPDIR/ecs_services.json"
  fi

  ecs_status=$(echo "$ecs_result" | python3 -c "import json,sys; print(json.load(sys.stdin).get('status','unknown'))" 2>/dev/null)
  if [[ "$ecs_status" == "degraded" || "$ecs_status" == "unhealthy" ]]; then
    [[ "$OVERALL" != "unhealthy" ]] && OVERALL="degraded"
  fi
fi

# ---------------------------------------------------------------------------
# 4. ALB Target Group Health
# ---------------------------------------------------------------------------
alb_result=$(python3 -c "
import json, subprocess, sys

tg_result = subprocess.run(
    ['aws', 'elbv2', 'describe-target-groups', '--region', '$REGION', '--output', 'json'],
    capture_output=True, text=True
)
if tg_result.returncode != 0:
    print(json.dumps({'status': 'error', 'error': 'Failed to query target groups'}))
    sys.exit(0)

tgs = json.loads(tg_result.stdout).get('TargetGroups', [])
result = {'status': 'healthy', 'target_groups': {}}
issues = []

for tg in tgs:
    name = tg['TargetGroupName']
    arn = tg['TargetGroupArn']

    health_result = subprocess.run(
        ['aws', 'elbv2', 'describe-target-health', '--target-group-arn', arn,
         '--region', '$REGION', '--output', 'json'],
        capture_output=True, text=True
    )
    if health_result.returncode != 0:
        result['target_groups'][name] = {'status': 'error', 'error': 'Query failed'}
        issues.append(f'{name}: failed to query health')
        continue

    targets = json.loads(health_result.stdout).get('TargetHealthDescriptions', [])
    healthy_count = sum(1 for t in targets if t['TargetHealth']['State'] == 'healthy')
    unhealthy = [t for t in targets if t['TargetHealth']['State'] not in ('healthy', 'draining', 'initial')]
    draining = sum(1 for t in targets if t['TargetHealth']['State'] == 'draining')
    total = len(targets)

    tg_status = 'healthy' if healthy_count > 0 else ('empty' if total == 0 else 'unhealthy')

    result['target_groups'][name] = {
        'healthy': healthy_count,
        'draining': draining,
        'unhealthy': len(unhealthy),
        'total': total,
        'status': tg_status
    }

    if unhealthy:
        for u in unhealthy:
            target_id = u['Target']['Id']
            state = u['TargetHealth']['State']
            reason = u['TargetHealth'].get('Reason', 'unknown')
            issues.append(f'{name}: {target_id} is {state} ({reason})')

    if total > 0 and healthy_count == 0:
        issues.append(f'{name}: no healthy targets ({total} total)')

if issues:
    result['status'] = 'degraded'
    result['issues'] = issues

print(json.dumps(result))
" 2>/dev/null)

if [[ -n "$alb_result" ]]; then
  echo "$alb_result" > "$TMPDIR/alb_targets.json"
else
  echo '{"status":"error","error":"Failed to check ALB targets"}' > "$TMPDIR/alb_targets.json"
fi

alb_status=$(echo "$alb_result" | python3 -c "import json,sys; print(json.load(sys.stdin).get('status','unknown'))" 2>/dev/null)
if [[ "$alb_status" == "unhealthy" ]]; then
  OVERALL="unhealthy"
elif [[ "$alb_status" == "degraded" && "$OVERALL" != "unhealthy" ]]; then
  OVERALL="degraded"
fi

# ---------------------------------------------------------------------------
# 5. SSL Certificate Check (if domain is set)
# ---------------------------------------------------------------------------
if [[ -n "$DOMAIN" ]] && command -v openssl &>/dev/null; then
  ssl_expiry=$(echo | openssl s_client -servername "$DOMAIN" -connect "$DOMAIN:443" 2>/dev/null | openssl x509 -noout -enddate 2>/dev/null | cut -d= -f2)
  if [[ -n "$ssl_expiry" ]]; then
    ssl_epoch=$(date -d "$ssl_expiry" +%s 2>/dev/null || echo "0")
    ssl_days_left=$(( (ssl_epoch - NOW) / 86400 ))

    if [[ $ssl_days_left -lt 7 ]]; then
      echo '{"status":"critical","expires":"'"$ssl_expiry"'","days_left":'"$ssl_days_left"'}' > "$TMPDIR/ssl.json"
      OVERALL="unhealthy"
    elif [[ $ssl_days_left -lt 30 ]]; then
      echo '{"status":"warning","expires":"'"$ssl_expiry"'","days_left":'"$ssl_days_left"'}' > "$TMPDIR/ssl.json"
    else
      echo '{"status":"healthy","expires":"'"$ssl_expiry"'","days_left":'"$ssl_days_left"'}' > "$TMPDIR/ssl.json"
    fi
  else
    echo '{"status":"error","error":"Could not retrieve SSL certificate"}' > "$TMPDIR/ssl.json"
  fi
fi

# ---------------------------------------------------------------------------
# Build Final JSON
# ---------------------------------------------------------------------------
FINAL_JSON=$(python3 -c "
import json, os, glob

tmpdir = '$TMPDIR'
checks = {}
for f in glob.glob(os.path.join(tmpdir, '*.json')):
    name = os.path.splitext(os.path.basename(f))[0]
    with open(f) as fh:
        checks[name] = json.load(fh)

result = {
    'timestamp': '$NOW_ISO',
    'epoch': $NOW,
    'overall': '$OVERALL',
    'cluster': '$CLUSTER',
    'region': '$REGION',
    'checks': checks
}

# Include domain if configured
domain = '$DOMAIN'
if domain:
    result['domain'] = domain

print(json.dumps(result, indent=2))
" 2>/dev/null)

# Write status file
echo "$FINAL_JSON" > "$STATUS_FILE"

# ---------------------------------------------------------------------------
# Output
# ---------------------------------------------------------------------------
if $JSON_ONLY; then
  echo "$FINAL_JSON"
  [[ "$OVERALL" == "healthy" ]] && exit 0 || exit 1
fi

# Build human-readable summary
if ! $QUIET && ! $JSON_ONLY; then
  if [[ "$OVERALL" == "healthy" ]]; then
    text="✅ ECS health check ($CLUSTER): all systems healthy"
  else
    issues=$(echo "$FINAL_JSON" | python3 -c "
import json, sys
data = json.load(sys.stdin)
issues = []
for name, check in data['checks'].items():
    if isinstance(check, dict) and check.get('status') not in ('healthy', 'scaled_down'):
        if 'issues' in check:
            issues.extend(check['issues'])
        elif 'error' in check:
            issues.append(f'{name}: {check[\"error\"]}')
        else:
            issues.append(f'{name}: {check.get(\"status\", \"unknown\")}')
print('; '.join(issues[:5]) if issues else '$OVERALL')
" 2>/dev/null)

    if [[ "$OVERALL" == "degraded" ]]; then
      text="⚠️  ECS health ($CLUSTER): DEGRADED — $issues"
    else
      text="🔴 ECS health ($CLUSTER): UNHEALTHY — $issues"
    fi
  fi

  echo "$text"
fi

# Write alert marker on failure (for external consumers)
if ! $QUIET && [[ "$OVERALL" != "healthy" ]]; then
  alert_file="${OUTDIR}/ecs-health-alert.json"
  echo "{\"timestamp\":\"$NOW_ISO\",\"cluster\":\"$CLUSTER\",\"overall\":\"$OVERALL\"}" > "$alert_file"

  # Auto-diagnose via log analyzer if available
  SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
  if [[ -x "$SCRIPT_DIR/cloudwatch-logs.sh" ]]; then
    LOG_ANALYSIS=$(ECS_CLUSTER="$CLUSTER" ECS_REGION="$REGION" ECS_SERVICES="${ECS_SERVICES:-}" \
      ECS_HEALTH_STATE="$STATUS_FILE" ECS_HEALTH_OUTDIR="$OUTDIR" \
      "$SCRIPT_DIR/cloudwatch-logs.sh" auto-diagnose --minutes 30 2>/dev/null)
    if [[ -n "$LOG_ANALYSIS" ]]; then
      python3 -c "
import json, sys
with open('$alert_file') as f:
    alert = json.load(f)
alert['log_analysis'] = sys.stdin.read().strip()
with open('$alert_file', 'w') as f:
    json.dump(alert, f, indent=2)
" <<< "$LOG_ANALYSIS" 2>/dev/null || true
    fi
  fi
fi

[[ "$OVERALL" == "healthy" ]] && exit 0 || exit 1


Security scan verdicts — ClawScan: suspicious | static analysis: clean | VirusTotal: suspicious | SkillSpector: clean (LOW)