Skill: vahagn-madatyan/cisco-device-health (version 1.0.0)

--- name: cisco-device-health description: >- Cisco IOS-XE and NX-OS device health check and triage procedure. Use when troubleshooting Cisco routers, switches, or Nexus platforms — assessing CPU, memory, interfaces, routing, and environment. Covers both IOS-XE (ISR, ASR, Catalyst 9K) and NX-OS (Nexus 3K/5K/7K/9K) with platform-specific commands, thresholds, and decision trees that account for IOS-XE QFP/RP architecture and NX-OS VDC isolation. license: Apache-2.0 metadata: safety: read-only author: network-security-skills-suite version: "1.0.0" openclaw: '{"emoji":"🔍","safetyTier":"read-only","requires":{"bins":["ssh"],"env":[]},"tags":["cisco","health","triage","nx-os"],"mcpDependencies":[],"egressEndpoints":[]}' --- # Cisco IOS-XE / NX-OS Device Health Check Structured triage procedure for assessing Cisco device health across IOS-XE and NX-OS platforms. Produces a prioritized findings report with severity classifications and recommended actions. Commands are labeled **[IOS-XE]** or **[NX-OS]** where they diverge. Unlabeled commands work on both platforms. ## When to Use - Device reported as slow, dropping traffic, or unresponsive - Scheduled health audit of IOS-XE routers/switches or Nexus platforms - Post-change verification after upgrades, patches, or configuration changes - Capacity planning data collection for CPU, memory, and link utilization - Incident response when a Cisco device is suspected as the fault domain - NX-OS VDC isolation troubleshooting — health check scoped to a specific VDC ## Prerequisites - SSH or console access to the device (privilege 1 minimum for IOS-XE, `show` role for NX-OS) - IOS-XE 16.x/17.x or NX-OS 9.x/10.x (commands validated against IOS-XE 17.3+ and NX-OS 10.2+) - Network reachability to management interface confirmed - Awareness of the device's normal baseline (CPU, memory, traffic patterns) - For NX-OS: identify which VDC you are operating in (`show vdc current`) ## Procedure Follow this sequence. Each step produces data for the final report. If the device is unresponsive, skip to Step 6 for crash recovery. ### Step 1: Establish Baseline Context Collect identity, uptime, and platform context. **[IOS-XE]** ``` show version | include uptime|Version|bytes of memory show inventory | include PID show clock ``` **[NX-OS]** ``` show version | include uptime|system version|Memory show inventory | include PID show clock show vdc current ``` Record: hostname, OS version, uptime, hardware model, current time. Short uptime means a recent reload or crash — investigate immediately. On NX-OS, confirm VDC context: all subsequent commands apply to this VDC only. ### Step 2: CPU Utilization IOS-XE separates Route Processor (RP) and QuantumFlow Processor (QFP). Triage both planes independently. NX-OS uses a unified supervisor model — `show system resources` is the primary view. **[IOS-XE]** ``` show processes cpu sorted | head 20 show processes cpu history show platform software status control-processor brief show platform hardware qfp active statistics drop ``` **[NX-OS]** ``` show system resources show processes cpu sort | head 20 show module ``` **IOS-XE interpretation:** Compare 5-second, 1-minute, and 5-minute RP averages. If RP CPU is normal but QFP drops are climbing, the data plane is overloaded while control plane is healthy — different remediation than RP overload. Key RP processes: IP Input (traffic storms), BGP Router (route churn), Crypto IKMP (VPN storms), IOSD (control plane congestion). **NX-OS interpretation:** `show system resources` reports CPU as kernel + user percentages. Normal idle should be above 60%. NX-OS control plane CPU runs hotter than IOS-XE RP because the supervisor handles both planes. Key processes: `netstack` (forwarding), `bgp` (route processing), `stp` (spanning tree), `vpc` (vPC keepalive). ### Step 3: Memory Utilization **[IOS-XE]** ``` show memory statistics show processes memory sorted | head 15 ``` **[NX-OS]** ``` show system resources show processes memory | head 15 ``` **IOS-XE:** Calculate used percentage from processor pool: `(Total - Free) / Total * 100`. Check fragmentation: if largest free block < 10% of total free, memory is fragmented. **NX-OS:** `show system resources` reports memory as total/used/free in KB. NX-OS reserves a larger kernel memory footprint than IOS-XE — used percentages above 60% are typical. Watch for `MemFree` dropping below 15% of total. ### Step 4: Interface Health ``` show interfaces summary show interfaces counters errors ``` **[IOS-XE]** ``` show interfaces | include line protocol|drops|error|CRC show controllers [name] ``` **[NX-OS]** ``` show interface [name] | include errors|drops|CRC show hardware internal errors module [n] ``` For each interface with errors: - Calculate error rate: `errors / (input + output packets) * 100` - CRC errors → Layer 1 issue (cabling, optics, SFP seating) - Input errors without CRC → buffer overruns - Output drops → congestion; check QoS policy or link capacity - On NX-OS, `show hardware internal errors` reveals ASIC-level drops not visible in standard interface counters ### Step 5: Routing Protocol Health ``` show ip route summary ``` **[IOS-XE]** ``` show ip bgp summary show ip ospf neighbor show ip eigrp neighbors ``` **[NX-OS]** ``` show bgp ipv4 unicast summary show ip ospf neighbors show ip eigrp neighbors show vpc brief ``` Verify: expected route count, all neighbors in established/full state, no unexpected withdrawals. On NX-OS, vPC peer-link state is critical — a failed peer-link orphans traffic. BGP prefix counts that deviate >10% from baseline indicate route churn. ### Step 6: Environment and Platform **[IOS-XE]** ``` show environment all show platform software status control-processor brief show logging | include %|Error|Warning|traceback | tail 50 dir crashinfo: ``` **[NX-OS]** ``` show environment show module show logging last 50 show cores ``` Check: power supply status, fan health, temperature readings. Any environmental alarm escalates immediately. On IOS-XE, review crashinfo: for recent crash dumps. On NX-OS, `show cores` lists supervisor and module core dumps — presence indicates a process crash requiring investigation. ## Threshold Tables Reference: `references/threshold-tables.md` for detailed per-parameter thresholds. | Parameter | Normal | Warning | Critical | Platform Notes | |-----------|--------|---------|----------|----------------| | CPU 5-min avg | < 40% | 40–70% | > 70% | NX-OS: expect 5–10% higher baseline | | CPU 5-sec spike | < 80% | 80–90% | > 90% | IOS-XE RP only; NX-OS: use system resources | | QFP drops/sec | < 100 | 100–1000 | > 1000 | IOS-XE only | | Memory used | < 70% | 70–85% | > 85% | NX-OS: normal baseline is 55–65% | | Interface error rate | < 0.01% | 0.01–0.1% | > 0.1% | Both platforms | | Output drops/hr | < 100 | 100–1000 | > 1000 | Both platforms | | Routing neighbors | All established | Flapping | Down | NX-OS: include vPC state | | Temperature | Within spec | 5°C of max | At/above max | Per-sensor spec varies | ## Decision Trees ### Primary Triage ``` Is the device reachable? ├── No → Check console, power, environment. Collect crashinfo after recovery. └── Yes ├── Identify platform → IOS-XE or NX-OS? │ ├── [IOS-XE] CPU issue? │ ├── RP CPU high, QFP normal → Control plane overload │ │ ├── BGP Router high → Route churn, check peer stability │ │ ├── IP Input high → Traffic storm, check ACLs/CoPP │ │ └── Other process → Collect 'show tech-support' for TAC │ ├── RP normal, QFP drops high → Data plane overload │ │ └── Check traffic rates, ACL complexity, NAT sessions │ └── Both high → Platform capacity exceeded, reduce load │ ├── [NX-OS] CPU issue? │ ├── Check VDC context → Are you in the correct VDC? │ ├── System resources CPU high │ │ ├── netstack high → Forwarding overload, check ARP/routes │ │ ├── bgp high → Route churn, check table size and peers │ │ ├── stp/vpc high → L2 instability, check topology │ │ └── Other → Collect 'show tech-support' for TAC │ └── Per-module issue → Specific linecard problem │ └── show module → Identify failed/degraded module │ ├── Memory issue? (either platform) │ ├── [IOS-XE] Fragmentation high → Schedule reload during window │ ├── [NX-OS] MemFree < 15% → Identify top consumers │ └── Steady growth → Memory leak, collect allocation data, plan reload │ ├── Interface errors? → Classify error type │ ├── CRC/input errors → Layer 1 (cable, optic, SFP) │ ├── Output drops → QoS or congestion │ └── [NX-OS] ASIC errors → Hardware issue, check module health │ └── All within thresholds → Document clean health ``` ### Escalation Criteria Escalate to senior engineer or TAC when: - CPU sustained > 90% for 15+ minutes with no identifiable cause - Memory below 15% free with no recent change to explain consumption - Traceback, CPUHOG, or MALLOCFAIL in logs (IOS-XE); core dump present (NX-OS) - Any environmental alarm (power, fan, temperature) - More than 3 routing neighbor state changes in the last hour - NX-OS module in non-OK state or vPC peer-link failure - QFP drops exceeding 10,000/sec with no traffic anomaly explanation (IOS-XE) ## Report Template ``` DEVICE HEALTH REPORT ==================== Device: [hostname] Platform: [IOS-XE | NX-OS] Model: [PID from inventory] Software: [version] Uptime: [uptime string] VDC: [VDC name — NX-OS only, omit for IOS-XE] Check Time: [timestamp] Performed By: [operator/agent] SUMMARY: [HEALTHY | WARNING | CRITICAL | EMERGENCY] FINDINGS: 1. [Severity] [Component] — [Description] Platform: [IOS-XE | NX-OS] Observed: [metric value] Threshold: [normal/warning/critical range] Action: [recommended action] 2. ... PLATFORM-SPECIFIC NOTES: - [IOS-XE: QFP drop summary or NX-OS: VDC/module status] RECOMMENDATIONS: - [Prioritized action list] NEXT CHECK: [date based on severity — CRITICAL: 24hr, WARNING: 7d, HEALTHY: 30d] ``` ## Troubleshooting ### Device Unresponsive to SSH Try console access. If console also unresponsive, check power and environment via out-of-band management (CIMC for UCS-based, smart PDU, or console server). On IOS-XE: collect `dir crashinfo:` after recovery. On NX-OS: `show cores` and `show system reset-reason`. ### NX-OS VDC Context Confusion Health data is VDC-scoped. If metrics look wrong, verify VDC: `show vdc current`. Switch VDC with `switchto vdc [name]`. The default VDC contains system-level resources; admin VDC shows aggregate hardware. ### CPU Spikes During Health Check Show commands and SNMP polling can briefly spike CPU. Wait 30 seconds after connecting before collecting CPU data. Use `terminal length 0` (IOS-XE) or `terminal length 0` (NX-OS) to avoid paging pauses. ### IOS-XE QFP Drops vs RP CPU High QFP drops with normal RP CPU means the data plane is stressed but control plane is healthy. Do not restart the device — the RP is functioning normally. Investigate traffic patterns, ACL complexity, and NAT session counts. Use `show platform hardware qfp active feature` to identify which QFP feature is consuming resources. ### NX-OS Module Issues If `show module` reports a module not in "ok" state, collect `show module internal exception-log module [n]` before taking action. Powercycling a linecard (`poweroff module [n]` / `no poweroff module [n]`) can resolve transient failures but requires change control. ### Inconsistent Memory Between Platforms IOS-XE and NX-OS report memory differently. IOS-XE `show memory statistics` shows IOS memory pools; NX-OS `show system resources` shows Linux-level memory. NX-OS baseline memory usage is higher (55–65% is normal) because the Linux kernel and system daemons consume more than IOS-XE's monolithic process model.

path: references/cli-reference.md
content: # Cisco IOS-XE / NX-OS CLI Reference

Commands organized by subsystem with IOS-XE and NX-OS side by side.
All commands are read-only (show/display only). IOS-XE validated against 17.3+,
NX-OS validated against 10.2+.

## CPU Subsystem

| Subsystem | IOS-XE Command | NX-OS Command | Key Output |
|-----------|---------------|---------------|------------|
| CPU summary | `show processes cpu sorted` | `show system resources` | 5sec/1min/5min avg (IOS-XE); kernel/user/idle % (NX-OS) |
| CPU history | `show processes cpu history` | *(not available)* | 72-hour ASCII trend graph |
| Top processes | `show processes cpu sorted \| head 20` | `show processes cpu sort \| head 20` | Per-process CPU consumption |
| Platform CPU | `show platform software status control-processor brief` | `show module` | RP load avg (IOS-XE); per-module status (NX-OS) |
| Data plane | `show platform hardware qfp active statistics drop` | `show hardware internal errors module [n]` | QFP drops (IOS-XE); ASIC errors (NX-OS) |

### IOS-XE CPU Notes

- `show processes cpu sorted` reports RP CPU only. Data plane (QFP) health requires
  `show platform hardware qfp active statistics drop`.
- The 5-second average includes interrupt-level CPU. High interrupt % with low
  process % indicates traffic-driven load (punt path).
- `show platform software status control-processor brief` gives per-core breakdown
  on multi-core RPs — useful for identifying single-core bottlenecks.

### NX-OS CPU Notes

- `show system resources` is the primary CPU command. Reports CPU as kernel%, user%,
  idle%. NX-OS has no direct equivalent to IOS-XE's `show processes cpu history`.
- NX-OS control plane typically runs 5–10% higher CPU at idle than IOS-XE due to
  Linux kernel overhead and system daemon activity.
- Per-module health via `show module` — each linecard has its own CPU/memory.
  A failing module may not impact supervisor CPU metrics.

## Memory Subsystem

| Subsystem | IOS-XE Command | NX-OS Command | Key Output |
|-----------|---------------|---------------|------------|
| Memory summary | `show memory statistics` | `show system resources` | Total/used/free pools (IOS-XE); MemTotal/MemUsed/MemFree in KB (NX-OS) |
| Top consumers | `show processes memory sorted \| head 15` | `show processes memory \| head 15` | Per-process allocation |
| Platform memory | `show memory platform information` | `show system internal mts buffers summary` | Per-region DRAM/SRAM (IOS-XE); MTS buffers (NX-OS) |
| Leak detection | `show memory dead` | `show processes memory` (watch over time) | Dead/unreachable blocks (IOS-XE) |
| Fragmentation | `show memory statistics` (Largest Free field) | *(not applicable — Linux manages)* | Largest contiguous free block |

### IOS-XE Memory Notes

- IOS memory pools (processor, I/O) are separate from Linux-level memory.
  `show memory statistics` shows IOS pools; `show platform software process slot R0 monitor` shows Linux.
- Fragmentation matters on IOS-XE: if largest free block < 10% of total free,
  allocations can fail even with free memory available (MALLOCFAIL).
- `show memory dead` reveals allocated-but-unreachable blocks — a definitive leak indicator.

### NX-OS Memory Notes

- NX-OS reports Linux-style memory. Normal baseline usage is 55–65% because the
  kernel, system daemons, and feature processes all reside in memory at boot.
- No concept of memory fragmentation as in IOS-XE — Linux's virtual memory
  manager handles page allocation differently.
- Watch MemFree trend: dropping below 15% of MemTotal warrants investigation.

## Interface Subsystem

| Subsystem | IOS-XE Command | NX-OS Command | Key Output |
|-----------|---------------|---------------|------------|
| Summary | `show interfaces summary` | `show interfaces summary` | One-line per-interface status |
| Error counters | `show interfaces counters errors` | `show interfaces counters errors` | Per-interface error counts |
| Detailed stats | `show interfaces [name]` | `show interface [name]` | Full counters, rates, drops |
| Hardware detail | `show controllers [name]` | `show hardware internal errors module [n]` | SFP DOM, PHY errors (IOS-XE); ASIC drops (NX-OS) |
| L1 diagnostics | `show controllers [name]` (DOM section) | `show interface [name] transceiver details` | Tx/Rx power, temperature, voltage |
| IP summary | `show ip interface brief` | `show ip interface brief` | IP, status, protocol |

### Interface Interpretation Notes

- Both platforms: CRC errors almost always mean Layer 1 (cabling, optics, SFP seating).
- IOS-XE: `show controllers` includes SFP DOM (Tx/Rx power, temp). Low Rx power = fiber/optic issue.
- NX-OS: `show interface transceiver details` provides DOM data. Also check
  `show hardware internal errors` for ASIC-level drops invisible to `show interface`.
- Output drops on both platforms indicate egress congestion — check QoS policy and link capacity.

## Routing Subsystem

| Subsystem | IOS-XE Command | NX-OS Command | Key Output |
|-----------|---------------|---------------|------------|
| Route summary | `show ip route summary` | `show ip route summary` | Per-protocol route counts |
| BGP peers | `show ip bgp summary` | `show bgp ipv4 unicast summary` | Neighbor state, prefix count |
| BGP detail | `show ip bgp neighbors [addr]` | `show bgp ipv4 unicast neighbors [addr]` | Session state, timers, messages |
| OSPF neighbors | `show ip ospf neighbor` | `show ip ospf neighbors` | Neighbor ID, state, dead time |
| EIGRP neighbors | `show ip eigrp neighbors` | `show ip eigrp neighbors` | Address, hold time, SRTT, Q count |
| vPC state | *(not applicable)* | `show vpc brief` | vPC domain, peer-link, keepalive |

### Routing Notes

- IOS-XE BGP: `State/PfxRcd` column showing a number = established. Any text = not established.
- NX-OS BGP: command syntax differs (`show bgp ipv4 unicast summary` not `show ip bgp summary`).
- NX-OS vPC: peer-link failure orphans traffic. `show vpc brief` → peer status must be "peer adjacency
  formed ok." Consistency check failures indicate config divergence between peers.
- OSPF: only `Full` state is healthy (or `2-Way` for DROther on broadcast networks).

## Environment and Platform

| Subsystem | IOS-XE Command | NX-OS Command | Key Output |
|-----------|---------------|---------------|------------|
| Environment | `show environment all` | `show environment` | Temp, power, fans |
| Platform status | `show platform` | `show module` | Module/slot state |
| RP status | `show platform software status control-processor brief` | `show system resources` | CPU, memory, load avg |
| Redundancy | `show redundancy` | `show system redundancy status` | Active/standby state |
| Syslog | `show logging \| tail 50` | `show logging last 50` | Recent log messages |
| Crash data | `dir crashinfo:` | `show cores` | Crash dump files |
| Reset reason | `show version \| include reason` | `show system reset-reason` | Last reload/reset cause |
| VDC context | *(not applicable)* | `show vdc current` | Current VDC identity |

### Environment Notes

- Both platforms: any power supply showing "Failed" or fan showing "Failed" is an emergency.
- IOS-XE: temperature thresholds are per-sensor and shown in `show environment all` output.
- NX-OS: `show module` is critical — a module not in "ok" state may silently drop traffic
  on its ports without affecting supervisor metrics.
- NX-OS: `show cores` lists core dumps from any process. Presence of recent cores warrants TAC case.


path: references/threshold-tables.md
content: # Cisco IOS-XE / NX-OS Threshold Tables

Detailed threshold definitions for device health parameters across both platforms.
Platform-specific baselines are noted where they differ. Adjust thresholds per
device role and traffic profile.

## CPU Thresholds

### RP CPU 5-Minute Average (IOS-XE)

| Level | Range | Meaning | Action |
|-------|-------|---------|--------|
| Normal | 0–40% | Typical operating range | No action |
| Warning | 41–70% | Elevated load | Identify top processes, monitor trend |
| Critical | 71–90% | High load, risk of protocol delays | Investigate top consumers, reduce load |
| Emergency | > 90% | Control plane at risk | Immediate intervention, collect show tech |

### Supervisor CPU (NX-OS)

NX-OS supervisor baseline runs 5–10% higher than IOS-XE RP due to Linux kernel and
daemon overhead. Adjust thresholds accordingly.

| Level | Range | Meaning | Action |
|-------|-------|---------|--------|
| Normal | 0–50% | Typical for NX-OS supervisor | No action |
| Warning | 51–75% | Elevated load | Identify top processes via `show processes cpu sort` |
| Critical | 76–90% | High load, risk of protocol timeouts | Investigate consumers, check for stuck processes |
| Emergency | > 90% | Supervisor at risk | Immediate triage, collect show tech-support |

### CPU 5-Second Spike (IOS-XE)

| Level | Range | Meaning | Action |
|-------|-------|---------|--------|
| Normal | 0–60% | Transient spikes expected | No action |
| Warning | 61–80% | Frequent spikes | Check for polling storms, route churn |
| Critical | 81–95% | Sustained spikes affecting protocol timers | Identify trigger, reduce load |
| Emergency | > 95% | CPU saturation | Risk of process crash, immediate triage |

### CPU Interrupt Percentage (IOS-XE)

| Level | Range | Meaning | Action |
|-------|-------|---------|--------|
| Normal | 0–20% | Normal packet processing | No action |
| Warning | 21–40% | Elevated punt rate | Verify expected traffic levels |
| Critical | 41–70% | Possible traffic storm or punted traffic | Check for loops, broadcast storms, CoPP |
| Emergency | > 70% | RP data plane overload | Apply CoPP or ACL to filter punted traffic |

### QFP Drop Rate (IOS-XE only)

| Level | Drops/sec | Meaning | Action |
|-------|-----------|---------|--------|
| Normal | < 100 | Normal operation | No action |
| Warning | 100–1,000 | Elevated drops | Identify drop reason via QFP feature stats |
| Critical | 1,000–10,000 | Significant data plane issues | Investigate ACL, NAT, or traffic anomalies |
| Emergency | > 10,000 | Data plane severely impaired | Immediate investigation, may need traffic diversion |

## Memory Thresholds

### Processor Memory Used (IOS-XE)

| Level | Range | Meaning | Action |
|-------|-------|---------|--------|
| Normal | 0–70% | Healthy memory utilization | No action |
| Warning | 71–80% | Elevated usage | Identify large consumers, monitor trend |
| Critical | 81–90% | Memory pressure, MALLOCFAIL risk | Check for leaks, consider feature reduction |
| Emergency | > 90% | Imminent MALLOCFAIL | Schedule reload, open TAC case |

### System Memory (NX-OS)

NX-OS baseline memory usage is 55–65% at steady state. Thresholds reflect this
higher baseline.

| Level | MemFree % | Meaning | Action |
|-------|-----------|---------|--------|
| Normal | > 30% | Healthy for NX-OS | No action |
| Warning | 20–30% | Elevated usage | Identify top consumers, monitor trend |
| Critical | 10–20% | Memory pressure | Investigate growth, check for leaks |
| Emergency | < 10% | Risk of OOM, process kills | Immediate investigation, plan reload |

### Memory Fragmentation Ratio (IOS-XE only)

Ratio of largest free block to total free memory. Not applicable on NX-OS
(Linux virtual memory manager handles paging).

| Level | Ratio | Meaning | Action |
|-------|-------|---------|--------|
| Normal | > 20% | Healthy allocation | No action |
| Warning | 10–20% | Moderate fragmentation | Monitor for degradation |
| Critical | 5–10% | Significant fragmentation | Schedule maintenance window reload |
| Emergency | < 5% | Severe — MALLOCFAIL imminent | Reload ASAP during next window |

## Interface Thresholds

### Error Rate (both platforms)

| Level | Rate | Meaning | Action |
|-------|------|---------|--------|
| Normal | < 0.001% | No significant errors | No action |
| Warning | 0.001–0.01% | Low error rate | Monitor, check optic levels |
| Critical | 0.01–0.1% | Significant errors | Investigate Layer 1 (cable/SFP) |
| Emergency | > 0.1% | High error rate | Replace cable/optic, check remote end |

### Output Drops Per Hour (both platforms)

| Level | Range | Meaning | Action |
|-------|-------|---------|--------|
| Normal | 0–50/hr | Minimal drops | No action |
| Warning | 51–500/hr | Moderate congestion | Review QoS policy, check bandwidth |
| Critical | 501–5,000/hr | Significant congestion | Tune QoS, consider link upgrade |
| Emergency | > 5,000/hr | Severe congestion | Immediate QoS intervention or upgrade |

### CRC Errors Per Hour (both platforms)

| Level | Range | Meaning | Action |
|-------|-------|---------|--------|
| Normal | 0 | Clean signal | No action |
| Warning | 1–10/hr | Intermittent L1 issue | Monitor, check SFP DOM readings |
| Critical | 11–100/hr | Active L1 problem | Replace cable, clean fiber, reseat SFP |
| Emergency | > 100/hr | Severe L1 failure | Replace cable and SFP, check patch panel |

### SFP Optical Power (both platforms)

| Level | Condition | Meaning | Action |
|-------|-----------|---------|--------|
| Normal | Within vendor spec | Clean optical link | No action |
| Warning | Within 2 dBm of low threshold | Degrading signal | Clean fiber ends, monitor |
| Critical | Below low warning threshold | Poor signal quality | Replace fiber run or SFP |
| Emergency | Below low alarm threshold | Link at risk | Immediate replacement |

## Routing Thresholds

### BGP Session Health (both platforms)

| Level | Condition | Meaning | Action |
|-------|-----------|---------|--------|
| Normal | Established, stable prefix count | Healthy peering | No action |
| Warning | Prefix count changed > 10% from baseline | Route churn | Investigate upstream |
| Critical | Session flap in last hour | Peer instability | Check transport, timers |
| Emergency | Session down > 5 minutes | Peering failure | Verify connectivity, peer status |

### OSPF Adjacency Health (both platforms)

| Level | Condition | Meaning | Action |
|-------|-----------|---------|--------|
| Normal | All neighbors Full | Healthy adjacencies | No action |
| Warning | Neighbor stuck in ExStart/Exchange > 30s | Slow formation | Check MTU, area config |
| Critical | Neighbor flap in last hour | Instability | Check interface, review timers |
| Emergency | Expected neighbor missing | Adjacency lost | Verify interface, check remote device |

### Route Table Size (both platforms)

| Level | Condition | Meaning | Action |
|-------|-----------|---------|--------|
| Normal | Within 10% of baseline | Stable | No action |
| Warning | 10–25% deviation | Moderate change | Verify expected changes |
| Critical | 25–50% deviation | Significant change | Investigate route source |
| Emergency | > 50% deviation or < 50% of baseline | Major event | Immediate investigation |

### vPC Health (NX-OS only)

| Level | Condition | Meaning | Action |
|-------|-----------|---------|--------|
| Normal | Peer adjacency formed, all vPCs consistent | Healthy | No action |
| Warning | Type-1 consistency check failure | Config divergence | Resolve config mismatch |
| Critical | Peer-link degraded or single member down | Reduced redundancy | Investigate link, plan repair |
| Emergency | Peer-link down or peer unreachable | Traffic orphaned | Immediate failover assessment |

## Environment Thresholds

### Temperature (both platforms)

| Level | Condition | Meaning | Action |
|-------|-----------|---------|--------|
| Normal | Below warning threshold (per sensor) | Within operating spec | No action |
| Warning | Within 5°C of shutdown threshold | Approaching limit | Check fans, airflow, room cooling |
| Critical | Above warning, below shutdown | Thermal stress | Fix cooling, reduce load if possible |
| Emergency | At or above shutdown threshold | Imminent shutdown | Emergency cooling, power down modules |

### Power Supply (both platforms)

| Level | Condition | Meaning | Action |
|-------|-----------|---------|--------|
| Normal | All PSUs OK, redundancy active | Fully protected | No action |
| Warning | Single PSU, no redundancy | Unprotected | Plan PSU installation |
| Critical | Redundant PSU failed | Lost redundancy | RMA failed PSU |
| Emergency | Single PSU degraded output | Power loss risk | Emergency maintenance |

### Fan Status (both platforms)

| Level | Condition | Meaning | Action |
|-------|-----------|---------|--------|
| Normal | All fans OK, RPM within spec | Adequate cooling | No action |
| Warning | Fan RPM low or degraded | Reduced cooling | Monitor temp, plan replacement |
| Critical | One fan failed | Thermal risk increasing | RMA fan tray, increase monitoring |
| Emergency | Multiple fans failed | Imminent thermal shutdown | Reduce load, emergency service |

### NX-OS Module Health

| Level | Condition | Meaning | Action |
|-------|-----------|---------|--------|
| Normal | All modules "ok" in `show module` | Fully operational | No action |
| Warning | Module diagnostics show minor failures | Degraded performance possible | Monitor, schedule diagnostics |
| Critical | Module in "powered-down" or error state | Ports on module are down | Investigate, attempt power cycle |
| Emergency | Module failure with core dump | Hardware or software fault | Collect diagnostics, open TAC case |


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (MEDIUM)