Skill: sdk-team/alibabacloud-ecs-reboot-or-crash-diagnosis (version 0.0.1)

--- name: alibabacloud-ecs-reboot-or-crash-diagnosis description: Diagnose ECS instance reboot or crash issues. First checks for abnormal maintenance events, then uses Cloud Assistant to check for internal restarts or kernel panics. Use this skill when users report ECS instance unexpected reboot, crash, abnormal shutdown, kernel panic, or OOM. Supports vmcore file analysis, kdump configuration, system log analysis, and Windows crash dump analysis. metadata: pattern: pipeline steps: "5" required_params: "instance_id, region_id" domain: aiops owner: ecs-team contact: [REDACTED_SECRET] ai-mode: disabled --- # ECS Instance Reboot/Crash Diagnosis Diagnose root cause of ECS instance unexpected reboot or crash. Uses standard workflow: check platform maintenance events first, then check internal system logs. Supports both Linux and Windows systems. ## Required Parameters Before starting diagnosis, **must** obtain the following parameters from user: | Parameter | Description | Example | |------|------|------| | `INSTANCE_ID` | ECS instance ID | `i-bp1a2b3c4d5e6f7g8h9j` | | `REGION_ID` | Region ID | `cn-hangzhou` | **If user does not provide any of the above parameters, must ask user first. Do not start diagnosis.** ## Mandatory Execution Rules 1. **Must obtain parameters first** — Instance ID and Region ID are required. Must ask user if missing. 2. **Standard workflow cannot be skipped** — Must execute in order: Maintenance Event Check → OSType Detection → System Log Check 3. **Must check Cloud Assistant status before diagnostics** — Before executing Step 3A/3B, must verify Cloud Assistant is running via `DescribeCloudAssistantStatus`. If not running, provide alternative diagnostic approaches. 4. **All diagnostic conclusions must be based on actual data** — No fabrication, speculation, or assumptions 5. **Output format must be strictly followed** — After diagnosis, **must read the complete template in `references/output-format.md`**, output strictly according to template structure. No free-form output, no omitted sections, no changed hierarchy. Every placeholder `{...}` in the template must be filled with actual data. --- ## Prerequisites ### CLI Tools - **aliyun-cli 3.3.3+** (required) — For calling Alibaba Cloud API - Installation & configuration: see [CLI Installation Guide](../../cli-installation-guide.md) ### AI-Mode Configuration (Required) Before using aliyun CLI commands, must configure AI-Mode: ```bash # Enable AI-Mode aliyun configure ai-mode enable # Set user-agent for skill identification aliyun configure ai-mode set-user-agent --user-agent [REDACTED_SECRET] # Update plugins aliyun plugin update ``` **After diagnosis complete, disable AI-Mode:** ```bash aliyun configure ai-mode disable ``` ### Alibaba Cloud Credentials Credentials must be pre-configured **outside of agent session**. Agent only verifies: ```bash aliyun configure list ``` ### Instance Requirements - **Cloud Assistant client must be installed and running** on the instance - Alibaba Cloud Linux: Pre-installed by default - Ubuntu/CentOS/Other: May require manual installation, check with `DescribeCloudAssistantStatus` API - Installation guide: https://help.aliyun.com/document_detail/64930.html - Instance status must be Running - **Note:** If Cloud Assistant is not running, diagnostic commands cannot be executed remotely. Must provide manual diagnostic steps to user. --- ## Required RAM Permissions See **[RAM Policies](references/ram-policies.md)** for the complete permission list and custom policy example. --- ## Step 1: Confirm Instance Information (Cannot Skip) **Verify instance exists and get basic information:** ```bash aliyun ecs describe-instances \ --biz-region-id <REGION_ID> \ --region <REGION_ID> \ --instance-ids '["<INSTANCE_ID>"]' ``` Confirm from returned JSON: - `RegionId` — Region ID (matches user provided) - `Status` — Instance status (Running/Stopped) - `InstanceName` — Instance name - `OSType` — Operating system type (**windows / linux**) **Record OSType for Step 3 branch selection.** --- ## Step 2: Check ECS Maintenance Events **Query instance historical system events to determine if platform maintenance caused reboot:** ```bash aliyun ecs describe-instance-history-events \ --biz-region-id <REGION_ID> \ --region <REGION_ID> \ --instance-id <INSTANCE_ID> \ --event-cycle-status Executed ``` **Event Analysis:** | Event Type | Meaning | Determination | Next Step | |---|---|---|---| | `SystemMaintenance.Reboot` | Reboot caused by system maintenance | Platform-initiated maintenance | Inform user, no further investigation needed | | `SystemFailure.Reboot` | Reboot caused by underlying hardware/system failure | Platform infrastructure failure | Suggest instance migration or contact support | | `InstanceFailure.Reboot` | Reboot caused by instance-level failure | **Instance internal issue detected by platform** | **Must continue to Step 3 for system log check** | | `InstanceExpiration.Stop` | Instance stopped due to expiration | Billing issue | Need renewal, no further investigation | | No relevant events | No platform maintenance events found | Not platform-initiated | Continue to Step 3 | **Important Notes for InstanceFailure.Reboot:** - This event indicates the platform detected an instance-level anomaly and triggered automatic recovery - Common causes: kernel panic, OOM, system hang, critical process failure - **Must execute Step 3** to check system logs for root cause - Even if no obvious errors in logs, the instance may have been unresponsive at kernel level **If maintenance event found:** - Clearly inform user of reboot cause (event type, time, reason) - Provide handling suggestions - End diagnosis flow **If no maintenance event found:** - Continue to Step 3, check internal system logs based on OSType --- ## Step 3A: Linux System Diagnosis (Execute when OSType is linux) ### Step 3A.1: Check Cloud Assistant Status (Mandatory) **Before executing diagnostic commands, verify Cloud Assistant is running:** ```bash aliyun ecs describe-cloud-assistant-status \ --biz-region-id <REGION_ID> \ --region <REGION_ID> \ --instance-id <INSTANCE_ID> ``` **Check the response:** ```json { "InstanceCloudAssistantStatusSet": { "InstanceCloudAssistantStatus": [ { "InstanceId": "i-xxx", "RegionId": "cn-xxx", "CloudAssistantStatus": "true", "LastHeartbeatTime": "2026-04-09T07:26:58Z" } ] } } ``` **Important Notes:** - `CloudAssistantStatus` is a **string** ("true"/"false"), not boolean - Check `LastHeartbeatTime` to ensure it's recent (within last few minutes) - Even if status is "true", RunCommand may still fail if service is unstable - **Always check RunCommand execution result** and handle failures gracefully - **Ubuntu vs RHEL differences:** - RHEL/CentOS/Alibaba Cloud Linux: Service name is `kdump`, crash files named `vmcore-*` - Ubuntu/Debian: Service name is `kdump-tools`, crash files named `dump.*` and `dmesg.*` - Diagnostic script now checks both service names and all crash file types **If CloudAssistantStatus is false or command fails:** - Cloud Assistant is not installed or not running on the instance - **Cannot proceed with remote diagnostic commands** - **Alternative approaches:** 1. Guide user to SSH into the instance and check logs manually 2. Provide manual diagnostic commands for user to execute 3. Suggest installing Cloud Assistant: [Installation Guide](https://help.aliyun.com/document_detail/64930.html) 4. Check instance monitoring data via CloudMonitor API **If CloudAssistantStatus is true:** - Proceed to Step 3A.2 ### Step 3A.2: Execute Linux Diagnostic Script Execute Linux diagnostic script via Cloud Assistant to check: - System reboot records (`last reboot`, `/var/log/messages` or `/var/log/syslog`) - Kernel Panic records (`dmesg`) - OOM records and `vm.panic_on_oom` configuration - Kdump configuration and crash dump file status - **Crash dump files**: vmcore (RHEL/CentOS) or dump.*/dmesg.* (Ubuntu/Debian) **Complete diagnostic commands: see [diagnostic-commands.md](references/diagnostic-commands.md#linux-system-diagnosis)** **Linux Result Analysis:** | Finding | Possible Cause | Suggestion | |---|---|---| | Kernel Panic + crash dump (vmcore/dump.*) | Kernel crash, dump file generated | Read dmesg.* file for panic reason, contact Alibaba Cloud technical support for deep analysis | | Kernel Panic + no crash dump | Kernel crash, but kdump not configured or not working | **Proceed to Step 5**: Recommend Kdump configuration for future crash capture | | OOM + panic_on_oom=1 | OOM triggered kernel panic | Disable panic_on_oom or increase memory | | OOM Killer | Memory insufficient causing process killed | Optimize memory usage or upgrade instance type | | SysRq triggered crash | Manual crash trigger via `/proc/sysrq-trigger` | Check if intentional test, review bash history and audit logs | | Normal reboot records | User or program triggered reboot | Check cron jobs or ops scripts | | No abnormal records | No system-level issues found | May be external factors, suggest monitoring | --- ## Step 3B: Windows System Diagnosis (Execute when OSType is windows) ### Step 3B.1: Check Cloud Assistant Status (Mandatory) **Before executing diagnostic commands, verify Cloud Assistant is running:** ```bash aliyun ecs describe-cloud-assistant-status \ --biz-region-id <REGION_ID> \ --region <REGION_ID> \ --instance-id <INSTANCE_ID> ``` **Check the response:** - `CloudAssistantStatus: true` — Cloud Assistant is running, proceed to Step 3B.2 - `CloudAssistantStatus: false` — Cloud Assistant is not running - **Cannot proceed with remote diagnostic commands** - Guide user to SSH/RDP into instance and run diagnostics manually - Suggest reinstalling Cloud Assistant: [Windows Installation Guide](https://help.aliyun.com/document_detail/64930.html) ### Step 3B.2: Execute Windows Diagnostic Script Execute Windows diagnostic script via Cloud Assistant to check: - System uptime and unexpected shutdown events (Event ID 41, 1074, 6008, 6006) - Memory dump configuration and pagefile settings - MEMORY.DMP and minidump files existence - BSOD events and application crashes **Complete diagnostic commands: see [diagnostic-commands.md](references/diagnostic-commands.md#windows-system-diagnosis)** **Windows Result Analysis:** | Finding | Possible Cause | Suggestion | |---|---|---| | Event 41 (Kernel-Power) | Unexpected shutdown/crash | Check for BSOD, dump files | | Dump configured + dump file exists | System crashed and captured dump | Contact Alibaba Cloud technical support for dump file analysis | | Dump configured + no dump file | Crash occurred but no dump captured | Check pagefile and disk space | | Dump not configured | Crash dumps disabled | Enable memory dump for diagnosis | | BSOD events found | Blue screen crash occurred | Check bug check code in dump | | No abnormal events | No system-level crash records | May be power issue or external factor | --- ## Step 3.5: Get Cloud Assistant Command Output (Required after Step 3) After executing diagnostic script via `RunCommand`, query the execution result: ```bash aliyun ecs describe-invocations \ --biz-region-id <REGION_ID> \ --region <REGION_ID> \ --instance-id <INSTANCE_ID> \ --invoke-id <INVOKE_ID> ``` **Important Notes:** - Use `--instance-id` (not `--instance-id.1`) for describe-invocations API - The `InvokeId` is returned by the `RunCommand` API call - Decode the `Output` field from Base64 to get diagnostic results - Check `InvokeStatus` to ensure command execution completed successfully --- ## Step 4: Analyze Crash Dump Files If Step 3 found crash dump files (vmcore on Linux, MEMORY.DMP/minidump on Windows), perform preliminary analysis. **Complete analysis commands: see [diagnostic-commands.md](references/diagnostic-commands.md#crash-dump-analysis)** > **Important:** If Linux vmcore files need deep analysis or Windows dump files (MEMORY.DMP/minidump) are found, recommend the user contact Alibaba Cloud technical support team for professional crash dump analysis assistance. --- ## Step 5: Recommend Kdump Configuration (If Not Configured) **If Step 3A found Kernel Panic records but no vmcore files, must advise user to configure Kdump.** ### When to Recommend Kdump Configuration - Kernel panic records found in dmesg or system logs, but `/var/crash` has no vmcore files - Kdump service status shows `inactive` or `failed` - `/proc/cmdline` does not contain `crashkernel=` parameter ### Key Points to Communicate 1. **Why Kdump is needed**: Without Kdump, kernel crashes will not generate vmcore files, making root cause analysis impossible. 2. **Configuration requirements**: - Reserve memory for crash kernel via `crashkernel=` kernel parameter - Enable and start the kdump (RHEL/CentOS) or kdump-tools (Ubuntu/Debian) service - Ensure sufficient disk space in `/var/crash` (or configured path) 3. **Configuration reference**: Provide guidance from [diagnostic-commands.md](references/diagnostic-commands.md#kdump-配置建议) ### Kdump Configuration Steps Summary **RHEL/CentOS/Alibaba Cloud Linux:** 1. Install: `yum install -y kexec-tools` 2. Add `crashkernel=auto` to kernel parameters in `/etc/default/grub` 3. Run `grub2-mkconfig -o /boot/grub2/grub.cfg` 4. Reboot the instance 5. Enable: `systemctl enable --now kdump` **Ubuntu/Debian:** 1. Install: `apt-get install -y kdump-tools` 2. Set `USE_KDUMP=1` in `/etc/default/kdump-tools` 3. Run `update-grub` (crashkernel parameter usually auto-added) 4. Reboot the instance 5. Verify: `systemctl status kdump-tools` ### Windows Memory Dump Configuration If Step 3B found BSOD events but no dump files: 1. Verify pagefile is configured and has sufficient size 2. Enable memory dump: System Properties → Advanced → Startup and Recovery → Settings 3. Select "Automatic memory dump" or "Kernel memory dump" 4. Ensure `CrashDumpEnabled` registry value is not 0 --- ## Final Output (Must execute after diagnosis complete) **After all diagnostic steps complete, must do both of the following:** 1. **Read `references/output-format.md`** — Get complete output format template 2. **Output strictly according to template structure** — Choose corresponding template based on actual result --- ## References - **[Output Format](references/output-format.md)** — Diagnostic result output template - **[Common Scenarios](references/scenarios.md)** — Typical problem diagnosis examples - **[Diagnostic Commands](references/diagnostic-commands.md)** — Complete diagnostic scripts and analysis commands

path: references/diagnostic-commands.md
content: # Diagnostic Commands Reference

本文档提供诊断检查项和命令参考。**Agent 应根据实际操作系统类型和版本，动态生成适配的诊断命令。**

> **重要原则**：
> - 不同 Linux 发行版的服务名称、日志路径、工具命令可能不同
> - 先通过 `DescribeInstances` 获取 OSType 和 OSName，再生成适配的命令
> - 命令应包含错误处理，避免因路径不存在或命令不可用而中断

---

## Linux 系统诊断

### 检查项清单

| 检查项 | 目的 | 参考命令 |
|--------|------|----------|
| 系统重启记录 | 查看历史重启时间和来源 | `last reboot`, `who -b` |
| 系统日志中的重启/关机记录 | 识别正常/异常关机 | `grep -i "reboot\|shutdown" /var/log/messages` 或 `/var/log/syslog` |
| Kernel Panic 记录 | 检测内核崩溃 | `dmesg | grep -i panic`, 日志文件搜索 |
| OOM Killer 记录 | 检测内存不足导致的进程终止 | 日志文件搜索 "Out of memory", "oom", "Kill process" |
| OOM Panic 配置 | 判断 OOM 是否会触发系统重启 | `sysctl -n vm.panic_on_oom` |
| Kdump 服务状态 | 验证崩溃转储是否配置并启用 | `systemctl status kdump` 或 `systemctl status kdump-tools` |
| Kdump 配置文件 | 获取转储文件存储路径 | `/etc/kdump.conf` 或 `/etc/default/kdump-tools` |
| 崩溃转储文件 | 检查是否存在 vmcore/dump 文件 | 检查配置的路径或默认 `/var/crash` |

### 操作系统差异对照表

| 项目 | RHEL/CentOS/Alibaba Cloud Linux | Ubuntu/Debian |
|------|--------------------------------|---------------|
| 系统日志路径 | `/var/log/messages` | `/var/log/syslog` |
| Kdump 服务名 | `kdump` | `kdump-tools` |
| Kdump 配置文件 | `/etc/kdump.conf` | `/etc/default/kdump-tools` |
| 崩溃转储文件名 | `vmcore` (目录: `127.0.0.1-date-time/`) | `dump.*` + `dmesg.*` |
| 默认转储路径 | `/var/crash` | `/var/crash` |

### 命令示例参考

以下命令仅供参考，Agent 应根据实际操作系统动态调整。

#### 1. 获取系统信息

```bash
# 操作系统版本
cat /etc/os-release

# 内核版本
uname -r

# 系统运行时间
uptime
```

#### 2. 系统重启历史

```bash
# 重启历史记录
last reboot | head -10

# 最近一次启动时间
who -b
```

#### 3. 系统日志检查

**RHEL/CentOS/Alibaba Cloud Linux:**
```bash
# 重启/关机相关日志
grep -i "reboot\|shutdown\|restart" /var/log/messages | tail -20

# Kernel Panic 记录
dmesg | grep -i "panic\|oops" | tail -20
grep -i "kernel panic" /var/log/messages | tail -10

# OOM 记录
grep -i "out of memory\|oom\|kill process" /var/log/messages | tail -20
```

**Ubuntu/Debian:**
```bash
# 重启/关机相关日志
grep -i "reboot\|shutdown\|restart" /var/log/syslog | tail -20

# Kernel Panic 记录
dmesg | grep -i "panic\|oops" | tail -20
grep -i "kernel panic" /var/log/syslog | tail -10

# OOM 记录
grep -i "out of memory\|oom\|kill process" /var/log/syslog | tail -20
```

#### 4. OOM Panic 配置

```bash
# 检查 OOM 时是否触发 panic
sysctl -n vm.panic_on_oom

# 返回值说明:
# 0 - OOM 只杀死进程，系统继续运行
# 1 - OOM 触发 kernel panic，导致系统重启
```

#### 5. Kdump 状态检查

**RHEL/CentOS/Alibaba Cloud Linux:**
```bash
# 检查 kdump 服务状态
systemctl status kdump

# 检查是否已配置
cat /etc/kdump.conf

# 获取转储路径 (默认 /var/crash)
grep "^path" /etc/kdump.conf
```

**Ubuntu/Debian:**
```bash
# 检查 kdump-tools 服务状态
systemctl status kdump-tools

# 检查是否已配置
cat /etc/default/kdump-tools

# 检查内核 crashkernel 参数
cat /proc/cmdline | grep crashkernel
```

#### 6. 崩溃转储文件检查

```bash
# 检查转储目录
ls -la /var/crash/

# RHEL/CentOS: 查找 vmcore 文件
find /var/crash -name "vmcore*" -type f -exec ls -lh {} \;

# Ubuntu/Debian: 查找 dump 和 dmesg 文件
find /var/crash -name "dump.*" -type f -exec ls -lh {} \;
find /var/crash -name "dmesg.*" -type f -exec ls -lh {} \;

# 检查最近 7 天的转储文件
find /var/crash -type f \( -name "vmcore*" -o -name "dump.*" -o -name "dmesg.*" \) -mtime -7
```

---

## Kdump 配置建议

### 何时需要建议配置 Kdump

以下情况应建议用户配置 Kdump：

1. **检测到 Kernel Panic 迹象但无 vmcore 文件**
   - dmesg 或系统日志中有 panic 记录
   - 但 `/var/crash` 目录为空或不存在

2. **Kdump 服务未运行**
   - `systemctl status kdump` 显示 inactive/failed
   - `systemctl status kdump-tools` 显示 inactive/failed

3. **内核未配置 crashkernel 参数**
   - `/proc/cmdline` 中没有 `crashkernel=` 参数

### Kdump 配置参考

**RHEL/CentOS/Alibaba Cloud Linux:**

```bash
# 1. 安装 kexec-tools (如未安装)
yum install -y kexec-tools

# 2. 配置 /etc/kdump.conf
# 默认配置通常可用，关键配置项：
# path /var/crash          # 转储文件存储路径
# core_collector makedumpfile -l --message-level 1 -d 31  # 压缩转储

# 3. 在 /etc/default/grub 的 GRUB_CMDLINE_LINUX 中添加 crashkernel 参数
# crashkernel=auto  或  crashkernel=128M

# 4. 更新 grub 配置
grub2-mkconfig -o /boot/grub2/grub.cfg

# 5. 重启系统使 crashkernel 生效
reboot

# 6. 启用并启动 kdump 服务
systemctl enable kdump
systemctl start kdump
systemctl status kdump
```

**Ubuntu/Debian:**

```bash
# 1. 安装 kdump-tools
apt-get install -y kdump-tools

# 2. 配置 /etc/default/kdump-tools
# USE_KDUMP=1

# 3. 更新 grub 配置 (安装时通常会自动添加 crashkernel 参数)
update-grub

# 4. 重启系统
reboot

# 5. 验证服务状态
systemctl status kdump-tools
```

### Kdump 配置验证

```bash
# 验证 crashkernel 参数已生效
cat /proc/cmdline | grep crashkernel

# 验证 kdump 服务状态
systemctl status kdump  # RHEL/CentOS
systemctl status kdump-tools  # Ubuntu/Debian
```

---

## Windows 系统诊断

### 检查项清单

| 检查项 | 目的 | 参考 PowerShell 命令 |
|--------|------|----------------------|
| 系统信息 | 获取 Windows 版本和主机名 | `Get-ComputerInfo` |
| 系统运行时间 | 判断最近是否重启过 | `[WMI]'\\.\root\cimv2:Win32_OperatingSystem'` |
| 意外关机事件 | 检测非正常关机 | Event ID 41, 6008, 6006, 1074 |
| 内存转储配置 | 验证是否配置了崩溃转储 | 注册表 `CrashControl` |
| 页面文件配置 | 转储文件需要页面文件支持 | `Get-CimInstance Win32_PageFileUsage` |
| MEMORY.DMP 文件 | 检查完整内存转储文件 | `Test-Path C:\Windows\MEMORY.DMP` |
| Minidump 文件 | 检查小型转储文件 | `Get-ChildItem C:\Windows\Minidump` |
| BSOD 事件 | 检测蓝屏错误报告 | WER 事件日志 |

### 事件 ID 说明

| Event ID | 来源 | 含义 |
|----------|------|------|
| 41 | Kernel-Power | 系统意外重启（未正常关机） |
| 1074 | User32 | 正常关机/重启，记录原因 |
| 6008 | EventLog | 上次关机是意外的 |
| 6006 | EventLog | 事件日志服务已停止（正常关机） |

### 内存转储类型

| CrashDumpEnabled | 类型 | 说明 |
|------------------|------|------|
| 0 | None | 禁用内存转储 |
| 1 | Complete | 完整内存转储（最大，约等于内存大小） |
| 2 | Kernel | 内核内存转储（中等大小） |
| 3 | Small | 小内存转储（64KB，Minidump） |
| 7 | Automatic | 自动内存转储（推荐） |

### PowerShell 命令示例

```powershell
# 系统信息
Get-ComputerInfo | Select-Object WindowsProductName, WindowsVersion, OsArchitecture, CsName

# 系统运行时间
$os = Get-CimInstance Win32_OperatingSystem
$uptime = (Get-Date) - $os.LastBootUpTime
Write-Host "Last boot: $($os.LastBootUpTime)"
Write-Host "Uptime: $($uptime.Days) days, $($uptime.Hours) hours"

# 意外关机事件
Get-WinEvent -FilterHashtable @{LogName="System"; ID=41,1074,6008,6006} -ErrorAction SilentlyContinue | 
    Select-Object TimeCreated, Id, Message -First 10

# 内存转储配置
$crashControl = Get-ItemProperty -Path "HKLM:\SYSTEM\CurrentControlSet\Control\CrashControl"
Write-Host "CrashDumpEnabled: $($crashControl.CrashDumpEnabled)"
Write-Host "DumpFile: $($crashControl.DumpFile)"
Write-Host "MinidumpDir: $($crashControl.MinidumpDir)"

# 页面文件配置
Get-CimInstance Win32_PageFileUsage | Select-Object Name, AllocatedBaseSize, CurrentUsage

# 检查 MEMORY.DMP
$dumpFile = $crashControl.DumpFile
if (-not $dumpFile) { $dumpFile = "C:\Windows\MEMORY.DMP" }
if (Test-Path $dumpFile) {
    $fileInfo = Get-Item $dumpFile
    Write-Host "MEMORY.DMP found: Size=$([math]::Round($fileInfo.Length/1GB,2)) GB, Modified=$($fileInfo.LastWriteTime)"
}

# 检查 Minidump 文件
$minidumpDir = $crashControl.MinidumpDir
if (-not $minidumpDir) { $minidumpDir = "C:\Windows\Minidump" }
if (Test-Path $minidumpDir) {
    Get-ChildItem -Path $minidumpDir -Filter "*.dmp" | Sort-Object LastWriteTime -Descending | Select-Object -First 5
}

# BSOD 事件
Get-WinEvent -FilterHashtable @{LogName="System"; ProviderName="Microsoft-Windows-WER-SystemErrorReporting"} -ErrorAction SilentlyContinue | 
    Select-Object TimeCreated, Id, Message -First 10
```

### Windows 内存转储配置建议

当检测到 BSOD 事件但无转储文件时，建议配置内存转储：

1. **通过系统属性配置**：
   - 右键"此电脑" → 属性 → 高级系统设置
   - 启动和故障恢复 → 设置
   - 选择"自动内存转储"或"内核内存转储"
   - 确保页面文件大小足够（至少内存大小 + 1MB）

2. **PowerShell 配置**：
```powershell
# 设置自动内存转储
Set-ItemProperty -Path "HKLM:\SYSTEM\CurrentControlSet\Control\CrashControl" -Name "CrashDumpEnabled" -Value 7

# 确保页面文件存在且大小足够
# 通常由系统自动管理，检查方法：
$cs = Get-CimInstance Win32_ComputerSystem
if ($cs.AutomaticManagedPagefile) {
    Write-Host "Pagefile is automatically managed"
} else {
    # 手动配置页面文件
    # 需要重启生效
}
```

---

## 崩溃转储文件分析

### Linux vmcore 分析

如果找到 vmcore 文件，可读取 `vmcore-dmesg.txt` 进行初步分析：

```bash
# 查看 vmcore-dmesg.txt 内容
cat /var/crash/127.0.0.1-*/vmcore-dmesg.txt

# 关键信息搜索
grep -i "kernel panic" /var/crash/*/vmcore-dmesg.txt
grep -i "RIP:" /var/crash/*/vmcore-dmesg.txt
grep -i "Call Trace" /var/crash/*/vmcore-dmesg.txt
```

**关键信息解读**：

| 关键字 | 含义 |
|--------|------|
| `Kernel panic - not syncing: VFS` | 文件系统相关问题 |
| `Kernel panic - not syncing: Attempted to kill init` | init 进程崩溃 |
| `Kernel panic - not syncing: Out of memory` | OOM 导致崩溃 |
| `RIP: 0010:` | 崩溃时的指令位置 |
| `Call Trace:` | 调用栈 |
| `MCE` / `Machine Check Exception` | 硬件错误 |

> **注意**：深度 vmcore 分析需要使用 `crash` 工具和调试符号包，建议联系阿里云技术支持获取专业分析。

### Windows 转储文件分析

使用 WinDbg 或 BlueScreenView 工具分析：

```
# WinDbg 命令
!analyze -v          # 自动分析崩溃原因
k                    # 查看调用栈
.bugcheck            # 查看 bugcheck 代码
```

> **注意**：深度 dump 分析建议联系阿里云技术支持。

---

## 通过云助手执行命令

诊断命令通过阿里云云助手远程执行：

### Linux 命令执行

```bash
aliyun ecs run-command \
  --biz-region-id <REGION_ID> \
  --region <REGION_ID> \
  --type RunShellScript \
  --instance-id <INSTANCE_ID> \
  --timeout 3600 \
  --command-content '<SCRIPT_CONTENT>'
```

### Windows 命令执行

```bash
aliyun ecs run-command \
  --biz-region-id <REGION_ID> \
  --region <REGION_ID> \
  --type RunPowerShellScript \
  --instance-id <INSTANCE_ID> \
  --timeout 3600 \
  --command-content '<SCRIPT_CONTENT>'
```

### 获取命令执行结果

```bash
aliyun ecs describe-invocations \
  --biz-region-id <REGION_ID> \
  --region <REGION_ID> \
  --instance-id <INSTANCE_ID> \
  --invoke-id <INVOKE_ID>
```

**注意**：`Output` 字段为 Base64 编码，需要解码后查看。


path: references/output-format.md
content: # Output Format Requirements

After diagnosis is complete, output results according to the following structure.

## Table of Contents

1. [Linux Diagnosis Result](#linux-diagnosis-result)
2. [Windows Diagnosis Result](#windows-diagnosis-result)

---

## Linux Diagnosis Result

```markdown
## Diagnostic Progress

### Step 1: Confirm Instance Information
> First need to confirm instance basic info and region.

- Instance ID: {instance_id}
- Region: {region_id}
- OS Type: Linux
- Current Status: {status}

### Step 2: Check Maintenance Events
> Check if platform maintenance events caused reboot.

**Findings:**
- {event_query_result}

### Step 3A: Linux System Diagnosis (if needed)
> No maintenance events found, checking internal restart or panic records.

**Cloud Assistant Status Check:**
- Cloud Assistant Running: {yes/no}
- If no: {explain why cannot proceed and provide alternative approaches}

**Findings:**
- {cloud_assistant_execution_result}

**Kdump Configuration Status:**
- Service Status: {active/inactive/active (kdump-tools)}
- Service Type: {kdump (RHEL/CentOS) / kdump-tools (Ubuntu/Debian)}
- Crash Dump Path: {configured_path}

**OOM Panic Configuration:**
- vm.panic_on_oom: {0/1}
- Impact: {OOM kills process only / OOM triggers kernel panic and reboot}

**Crash Dump File Check:**
- Found crash dumps: {yes/no}
- Dump type: {vmcore (RHEL) / dump.*+dmesg.* (Ubuntu)}
- Latest dump: {file_path, size, time}
- Panic reason (from dmesg): {panic_message_if_available}

**Alternative Diagnostic Approaches (if Cloud Assistant not available):**
```bash
# Provide these commands to user for manual execution via SSH
ssh root@{instance_public_ip}

# Check reboot history
last reboot

# Check system logs
grep -i "reboot\|shutdown\|panic\|oom" /var/log/syslog | tail -50

# Check dmesg for errors
dmesg | grep -i "panic\|oom\|error" | tail -20

# Check kdump
systemctl status kdump
ls -lh /var/crash/
```

### Step 4A: vmcore-dmesg.txt Analysis (if vmcore found)
> Found vmcore file, reading vmcore-dmesg.txt for preliminary analysis.

**Panic Reason:**
- {panic_specific_reason}

**Crash Location:**
- RIP: {function_and_address_at_crash}
- Involved Modules: {related_kernel_modules}

**Call Stack:**
```
{key_call_stack_fragment}
```

### Step 5: Kdump Configuration Recommendation (if no vmcore and kdump not configured)
> Kernel panic detected but no crash dump file found. Kdump is not properly configured.

**Current Kdump Status:**
- Service Status: {kdump_service_status}
- crashkernel Parameter: {present/absent in /proc/cmdline}
- Config File Exists: {yes/no}

**Why Kdump is Needed:**
Without Kdump configured, kernel crashes will not generate vmcore files, making root cause analysis impossible for future occurrences.

**Configuration Steps for {OS_Type}:**

{configuration_steps_based_on_os}

---

## Diagnostic Conclusion

- **Root Cause Analysis**: {root_cause}
- **Impact Scope**: {impact_scope}

---

## Recommendations

1. {recommendation_1}
2. {recommendation_2}
```

---

## Windows Diagnosis Result

```markdown
## Diagnostic Progress

### Step 1: Confirm Instance Information
> First need to confirm instance basic info and region.

- Instance ID: {instance_id}
- Region: {region_id}
- OS Type: Windows
- Current Status: {status}

### Step 2: Check Maintenance Events
> Check if platform maintenance events caused reboot.

**Findings:**
- {event_query_result}

### Step 3B: Windows System Diagnosis (if needed)
> No maintenance events found, checking Windows crash dump and event logs.

**System Uptime:**
- Last Boot Time: {last_boot_time}
- Uptime: {days} days, {hours} hours

**Unexpected Shutdown Events:**
- {shutdown_event_summary}

**Memory Dump Configuration:**
- CrashDumpEnabled: {0/1/2/3/7}
- Dump Type: {None/Complete/Kernel/Small/Automatic}
- Dump File Path: {dump_file_path}
- Pagefile: {configured/not configured}

**Memory Dump File Check:**
- Memory dump file: {found/not found}
- File size: {size}
- Last modified: {timestamp}

**Minidump Files:**
- Count: {count}
- Latest: {filename, timestamp}

**BSOD Events:**
- {bsod_event_summary}

---

## Diagnostic Conclusion

- **Root Cause Analysis**: {root_cause}
- **Impact Scope**: {impact_scope}

---

## Recommendations

1. {recommendation_1}
2. {recommendation_2}
```


path: references/ram-policies.md
content: # RAM 权限清单

本 Skill 执行所需的 RAM 权限（最小权限原则）:

## 必需权限

`ecs:DescribeInstances` — 确认实例存在并获取基本信息（状态、名称、操作系统类型）

`ecs:DescribeInstanceAttribute` — 获取实例详细属性，用于操作系统类型检测和分支选择

`ecs:DescribeInstanceHistoryEvents` — 查询实例历史维护事件，判断是否为平台触发重启

`ecs:DescribeCloudAssistantStatus` — 验证云助手运行状态，确保远程诊断命令可执行

`ecs:RunCommand` — 通过云助手执行诊断脚本（Linux Shell 或 Windows PowerShell）

`ecs:DescribeInvocations` — 获取云助手命令执行结果，提取诊断输出

## 权限说明

- **权限范围**: 仅包含诊断所需的只读和命令执行权限
- **写操作**: 无（本 Skill 不修改实例配置）
- **通配符**: 未使用（遵循最小权限原则）

## 自定义策略示例

```json
{
  "Version": "1",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ecs:DescribeInstances",
        "ecs:DescribeInstanceAttribute",
        "ecs:DescribeInstanceHistoryEvents",
        "ecs:DescribeCloudAssistantStatus",
        "ecs:RunCommand",
        "ecs:DescribeInvocations"
      ],
      "Resource": "*"
    }
  ]
}
```

## 使用场景

此权限配置适用于 ECS 实例故障诊断场景:
- 检查平台维护事件
- 通过云助手远程执行诊断命令
- 获取系统日志和崩溃转储文件信息
- 分析重启/崩溃根因


path: references/scenarios.md
content: # Common Diagnostic Scenarios

This document lists common diagnostic scenarios and expected outputs for the ecs-reboot-or-crash skill.

---

## Linux Scenarios

### Scenario 1: System Maintenance Reboot

```
Diagnosis Result:
- Found event: SystemMaintenance.Reboot
- Event time: 2025-03-20 10:00:00
- Reason: Planned system maintenance

Conclusion: Instance reboot was caused by Alibaba Cloud platform maintenance, which is normal ops activity.
```

---

### Scenario 2: Kernel Panic + vmcore Available

```
Diagnosis Result:
- No maintenance events found
- Cloud Assistant found: "Kernel panic - not syncing" record in dmesg
- Kdump service status: active
- Found vmcore: /var/crash/127.0.0.1-2025-03-20-10:30:00/vmcore (2.5G)
- vmcore time: 2025-03-20 10:30:00

vmcore-dmesg.txt analysis:
- Panic reason: Kernel panic - not syncing: Fatal exception in interrupt
- Crash location: RIP: 0010:nvme_queue_rq+0x1a2/0x4d0 [nvme]
- Involved module: nvme driver
- Call stack: nvme_queue_rq -> blk_mq_dispatch_rq_list -> ...

Conclusion: Instance rebooted due to NVMe driver abnormality causing kernel crash, vmcore dump file generated.
Suggestion: Check NVMe driver version, upgrade driver or kernel if needed. Use crash tool for deeper vmcore analysis.
```

---

### Scenario 3: Kernel Panic + No vmcore

```
Diagnosis Result:
- No maintenance events found
- Cloud Assistant found: "Kernel panic" record in dmesg
- Kdump service status: inactive
- Found vmcore: No

Conclusion: Instance rebooted due to kernel crash, but kdump not configured or not working, unable to capture vmcore.
Suggestion: Configure kdump service to capture vmcore on next crash for root cause analysis.
```

---

### Scenario 4: OOM with panic_on_oom Enabled

```
Diagnosis Result:
- No maintenance events found
- Cloud Assistant found: "Out of memory: Kill process" record in /var/log/messages
- vm.panic_on_oom: 1 (OOM triggers kernel panic)
- Kdump service status: active
- Found vmcore: Yes

Conclusion: OOM event triggered kernel panic because vm.panic_on_oom=1, causing system reboot.
Suggestions:
1. Disable panic_on_oom: sysctl -w vm.panic_on_oom=0 (add to /etc/sysctl.conf for persistence)
2. Optimize application memory usage or upgrade instance type
3. Review OOM killed processes to identify memory-hungry applications
```

---

### Scenario 5: OOM Killer Only

```
Diagnosis Result:
- No maintenance events found
- Cloud Assistant found: "Out of memory: Kill process" record in /var/log/messages
- vm.panic_on_oom: 0 (OOM only kills process, no panic)
- No kernel panic records found

Conclusion: Instance triggered OOM Killer due to insufficient memory, some processes were terminated but system continued running.
Suggestion: Optimize application memory usage, or upgrade instance type.
```

---

## Windows Scenarios

### Scenario 6: BSOD with Memory Dump

```
Diagnosis Result:
- No maintenance events found
- Unexpected shutdown event: Event ID 41 (Kernel-Power)
- BSOD events found in WER logs
- Memory dump configuration: Automatic memory dump (CrashDumpEnabled=7)
- Memory dump file found: C:\Windows\MEMORY.DMP (4.2 GB)
- Dump time: 2025-03-20 10:30:00

Conclusion: Windows BSOD crash occurred, memory dump captured.
Suggestion: Download MEMORY.DMP and analyze with WinDbg:
1. Install Windows Debugging Tools
2. Open dump file in WinDbg
3. Run: !analyze -v
```

---

### Scenario 7: BSOD without Dump (Not Configured)

```
Diagnosis Result:
- No maintenance events found
- Unexpected shutdown event: Event ID 41 (Kernel-Power)
- BSOD events found in WER logs
- Memory dump configuration: None (CrashDumpEnabled=0)
- Memory dump file: Not found

Conclusion: Windows BSOD crash occurred but memory dump was not configured.
Suggestions:
1. Enable memory dump: System Properties > Advanced > Startup and Recovery > Settings
2. Select "Automatic memory dump" or "Kernel memory dump"
3. Ensure pagefile is configured and has sufficient space
```

---

### Scenario 8: BSOD without Dump (Pagefile Issue)

```
Diagnosis Result:
- No maintenance events found
- Unexpected shutdown event: Event ID 41 (Kernel-Power)
- Memory dump configuration: Automatic memory dump (CrashDumpEnabled=7)
- Pagefile: Not configured
- Memory dump file: Not found

Conclusion: Windows BSOD crash occurred but memory dump was not captured because pagefile is not configured.
Suggestions:
1. Configure pagefile: System Properties > Advanced > Performance > Settings > Advanced > Virtual memory
2. Set pagefile size to at least RAM size + 1MB
3. Reboot for pagefile changes to take effect
```

---

### Scenario 9: Minidump Available

```
Diagnosis Result:
- No maintenance events found
- Unexpected shutdown event: Event ID 41 (Kernel-Power)
- Memory dump configuration: Small memory dump (CrashDumpEnabled=3)
- Minidump files found in C:\Windows\Minidump:
  - 032025-12345-01.dmp (128 KB, 2025-03-20 10:30:00)

Conclusion: Windows crash occurred, minidump captured.
Suggestion: Analyze minidump with WinDbg Preview (Microsoft Store) or BlueScreenView tool.
```

---

### Scenario 10: Application Crash Causing Instability

```
Diagnosis Result:
- No maintenance events found
- No unexpected shutdown events
- Application crash events found: Multiple crashes of {application_name}.exe
- No system crash dump files

Conclusion: Application crashes detected but no system-level crash. System remained running.
Suggestions:
1. Check application logs for crash details
2. Verify application compatibility with Windows version
3. Check for application updates or known issues
```


Security scan verdicts — ClawScan: suspicious | static analysis: clean | VirusTotal: clean | SkillSpector: clean (LOW)